So, buckle up because today, we\u2019re diving into how Azure SQL DB\u2019s vector capabilities can enable advanced resume matching. We\u2019ll be using Azure Document Intelligence and Azure OpenAI to make this magic happen. Let\u2019s get started!<\/p>\n
The Challenge: Finding the Perfect Candidate<\/strong><\/h2>\nImagine you\u2019re a recruiter with a stack of digital resumes taller than your inbox. You need to find candidates with specific skills and experiences, but manually reviewing each resume is time-consuming and prone to errors. You also need to know what focus areas to address in the interview if a candidate is a match but has some missing skills. What if there was a way to automate this process, ensure you never miss a great candidate, and make your interview process better? Enter Azure SQL DB\u2019s native vector capabilities!<\/p>\n
![\"funny](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/Designer-17.jpeg\")
<\/a><\/p>\n <\/p>\n
The Solution: Vector Similarity Search<\/strong><\/h2>\nBy leveraging Azure SQL DB\u2019s new vector<\/a> data type, we can store embeddings (vector representations) of resume content and perform similarity searches using built-in vector functions. This allows us to match resumes to job descriptions with incredible accuracy. Here\u2019s how we do it:<\/p>\nGetting Started<\/strong><\/h3>\nFor the purpose of this sample tutorial, we\u2019ll use a simple\u00a0Python notebook<\/a>\u00a0so that you can have everything up and running in less than 2 hours. Here\u2019s a quick overview of what you\u2019ll need to get started:<\/p>\nDataset<\/strong><\/h4>\nWe use a sample dataset from\u00a0Kaggle<\/a>\u00a0containing PDF resumes for this tutorial. For this tutorial, we will use 120 resumes from the\u00a0Information-Technology<\/strong>\u00a0folder<\/p>\nPrerequisites<\/strong><\/h4>\n\n- Azure Subscription<\/strong>:\u00a0Create one for free<\/a><\/span><\/li>\n
- Azure SQL Database<\/strong>:\u00a0Set up your database for free<\/a><\/span><\/li>\n
- Azure Document Intelligence<\/strong>\u00a0Create a Free Azure Document Intelligence resource<\/a><\/span><\/li>\n
- Azure OpenAI Access<\/strong>: Apply for access in the desired Azure subscription at\u00a0https:\/\/aka.ms\/oai\/access<\/a><\/span><\/li>\n
- Azure OpenAI Resource<\/strong>: Deploy an embeddings model (e.g.,\u00a0text-embedding-small\u00a0or\u00a0text-embedding-ada-002) and a\u00a0GPT-4.0\u00a0model for chat completion. Refer to the\u00a0resource deployment guide<\/a><\/span><\/li>\n
- Azure Data Studio<\/strong>: Download\u00a0here<\/a>\u00a0to manage your Azure SQL database and\u00a0execute the notebook<\/a><\/span><\/li>\n<\/ul>\n
Step 1: Extract & chunk content from PDF Resumes<\/strong><\/h3>\nFirst, we use Azure Document Intelligence to extract and chunk content from PDF resumes.<\/p>\n
Why Azure Document Intelligence?<\/strong><\/h4>\nAI Document Intelligence <\/a>is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately.\nThis means you can turn documents into usable data and shift your focus to acting on information rather than compiling it.<\/p>\nHere is an example of a Resume in PDF format from the dataset:\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/resume.png\")
<\/a><\/p>\nOne of the best parts about using Azure Document Intelligence is how easy it is to get started with prebuilt<\/strong> models. These models are designed to handle common document types like invoices, receipts, and more! Here\u2019s why you\u2019ll love using them:<\/p>\n\n- No Training Required:<\/strong> Prebuilt models<\/a> are ready to use out of the box & like having a team of document experts at your disposal. You don\u2019t need to spend time training them on your data.<\/span><\/li>\n
- High Accuracy:<\/strong> These models are built and maintained by Microsoft, ensuring high accuracy and reliability.<\/span><\/li>\n
- Easy Integration<\/strong>: With simple API calls, you can integrate these models into your applications quickly and efficiently.<\/span><\/li>\n
- Scalability:<\/strong> Azure\u2019s infrastructure ensures that your document processing can scale with your needs, handling everything from a few documents to millions.<\/span><\/li>\n<\/ul>\n
We define DocumentAnalysisClient\u00a0 to send requests to the Azure Document Intelligence<\/a>\u00a0service and receive responses containing the extracted text from the PDF resumes. It\u00a0provides operations for analyzing input documents using prebuilt and custom models through the\u00a0begin_analyze_document<\/code>\u00a0and\u00a0begin_analyze_document_from_url<\/code>\u00a0APIs.<\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/10\/docanalysisclient.png\")
<\/a><\/p>\nYou can also visualize how the document text extraction is performed using the\u00a0Document Intelligence Studio<\/a><\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/visualizedocintelligence.png\")
<\/a><\/p>\nWhy are we chunking? <\/strong><\/h4>\nWhen dealing with content that exceeds the embedding limit of our model, we need to break it into smaller pieces. This is because the model that generates embeddings can only handle a certain amount of text at a time.<\/p>\n
Here, we divide the extracted text into fixed chunks of 500 tokens. These chunks are later passed to the text-embedding-small<\/strong> model to generate text embeddings, as the model has an input token limit of 8192<\/strong><\/p>\nThis process breaks down resumes into manageable pieces, making them easier to analyze and process. Think of it as turning a giant pizza into bite-sized slices\u2014much easier to handle!<\/p>\n
Note<\/strong>: There are different ways of chunking<\/a> content, and the method you choose can depend on your specific needs and the structure of the text you\u2019re working with.<\/p>\nWe now have our Resumes extracted & chunked:<\/p>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/pdfchunkingoutput.png\")
<\/a><\/p>\nStep 2: Generating Embeddings<\/strong><\/h2>\nNext, we generate embeddings from the chunked content using the Azure OpenAI API.<\/a> These embeddings are vector representations of the text, capturing the semantic meaning of the content.<\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/get_embedding.png\")
<\/a><\/p>\nEach chunk now has a text embedding as seen below:<\/p>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/embeddings.png\")
<\/a><\/p>\nStep 3: Storing Embeddings in Azure SQL DB<\/strong><\/h3>\nWe then store these embeddings in Azure SQL DB using the new\u00a0Vector<\/a> data type. This allows us to perform similarity searches using built-in vector functions, enabling us to find the most suitable candidates based on their resumes.<\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/insertintosqldb.png\")
<\/a><\/p>\nWe now have our data in the SQL DB which we can use for Vector Search. This means that even if the exact words don\u2019t match, the underlying meaning can still be understood and matched effectively.<\/p>\n
![\"\"](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/10\/vectorstoredinsqldb.png\")
<\/a><\/p>\nFor example: If your search query is focused on project management skills, the embeddings can help find candidates with relevant experience, even if they use different terminology:<\/p>\n
Search Query: \u201cproject management\u201d<\/strong><\/p>\nMatched Candidate Skills:<\/strong><\/p>\n\n- \u201cLed multiple software development projects\u201d<\/li>\n
- \u201cCertified Scrum Master with experience in Agile methodologies\u201d<\/li>\n
- \u201cManaged cross-functional teams to deliver projects on time and within budget\u201d<\/li>\n<\/ul>\n
The embeddings capture the essence of \u201cproject management\u201d and match it with various related skills and experiences. It\u2019s like having a supercharged recruiter at your fingertips!<\/p>\n
Let’s now query our ResumeDocs table to get the top similar candidates given the User search query.<\/p>\n
Vector Search in SQL DB:<\/strong> Given any user search query, we can obtain the vector representation of that text. We then use this vector to calculate the cosine distance against all the resume embeddings stored in the database. By selecting only the closest matches, we can identify the resumes most relevant to the user\u2019s query. This helps in finding the most suitable candidates based on their resumes.<\/p>\nThe most common distance is the cosine similarity, which can be calculated quite easily in SQL with the help of the built-in vector distance functions.<\/p>\n
VECTOR_DISTANCE(‘distance metric’, V1, V2)<\/strong><\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/vectordistancefunction.png\")
<\/a><\/p>\nStep 4: Augmenting LLM Generation<\/strong><\/h3>\nFinally, we enhance language model generation with the Vector Search results from our Azure SQL Database. By informing a GPT-4o chat model with the results of our Vector search, we can provide rich, context-aware answers about candidates based on their resumes.<\/p>\n
The function\u00a0generate_completion<\/code> is defined to help ground the gpt-4o model with prompts and system instructions. Note that we are passing the results of the\u00a0vector_search_sql<\/code> we defined earlier<\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/10\/systempromptLLM.png\")
<\/a><\/p>\nWe can now create interactive loop where you can pose questions to the model and receive information grounded in your data.<\/p>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/LLMresponse.png\")
<\/a>\nLet’s look at Candidate 2:<\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/candidate2.png\")
<\/a>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/candidate3.png\")
<\/a><\/p>\nWe also can get a short summary at the end along with a Fun Microsoft fact about the technology we are hiring for as we had defined in the prompt:<\/p>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/12\/finalsummary.png\")
<\/a><\/p>\nHence by utilizing Azure SQL DB\u2019s<\/strong> vector capabilities along with the power of Azure Document Intelligence and Azure OpenAI, you can totally transform your hiring game.<\/p>\nNo more missing out on the perfect candidate, and your interviews will be more on point than ever. But why stop at resumes? Dive into your own documents\u2014whether they\u2019re project reports, research papers\u2014and uncover hidden gems. This cool approach not only saves you time but also boosts the quality of your recruitment and document analysis. Happy exploring and hiring! \ud83d\ude80<\/p>\n
Private Preview Sign-up<\/h2>\n
We are currently accepting requests from customers who would like to participate in the private preview and try out the Native Vector Support for SQL feature. If you are interested, please fill out\u00a0https:\/\/aka.ms\/azuresql-vector-eap<\/a><\/p>\n
<\/a><\/p>\n<\/p>\n
The Solution: Vector Similarity Search<\/strong><\/h2>\nBy leveraging Azure SQL DB\u2019s new vector<\/a> data type, we can store embeddings (vector representations) of resume content and perform similarity searches using built-in vector functions. This allows us to match resumes to job descriptions with incredible accuracy. Here\u2019s how we do it:<\/p>\nGetting Started<\/strong><\/h3>\nFor the purpose of this sample tutorial, we\u2019ll use a simple\u00a0Python notebook<\/a>\u00a0so that you can have everything up and running in less than 2 hours. Here\u2019s a quick overview of what you\u2019ll need to get started:<\/p>\nDataset<\/strong><\/h4>\nWe use a sample dataset from\u00a0Kaggle<\/a>\u00a0containing PDF resumes for this tutorial. For this tutorial, we will use 120 resumes from the\u00a0Information-Technology<\/strong>\u00a0folder<\/p>\nPrerequisites<\/strong><\/h4>\n\n- Azure Subscription<\/strong>:\u00a0Create one for free<\/a><\/span><\/li>\n
- Azure SQL Database<\/strong>:\u00a0Set up your database for free<\/a><\/span><\/li>\n
- Azure Document Intelligence<\/strong>\u00a0Create a Free Azure Document Intelligence resource<\/a><\/span><\/li>\n
- Azure OpenAI Access<\/strong>: Apply for access in the desired Azure subscription at\u00a0https:\/\/aka.ms\/oai\/access<\/a><\/span><\/li>\n
- Azure OpenAI Resource<\/strong>: Deploy an embeddings model (e.g.,\u00a0text-embedding-small\u00a0or\u00a0text-embedding-ada-002) and a\u00a0GPT-4.0\u00a0model for chat completion. Refer to the\u00a0resource deployment guide<\/a><\/span><\/li>\n
- Azure Data Studio<\/strong>: Download\u00a0here<\/a>\u00a0to manage your Azure SQL database and\u00a0execute the notebook<\/a><\/span><\/li>\n<\/ul>\n
Step 1: Extract & chunk content from PDF Resumes<\/strong><\/h3>\nFirst, we use Azure Document Intelligence to extract and chunk content from PDF resumes.<\/p>\n
Why Azure Document Intelligence?<\/strong><\/h4>\nAI Document Intelligence <\/a>is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately.\nThis means you can turn documents into usable data and shift your focus to acting on information rather than compiling it.<\/p>\nHere is an example of a Resume in PDF format from the dataset:\n<\/a><\/p>\nOne of the best parts about using Azure Document Intelligence is how easy it is to get started with prebuilt<\/strong> models. These models are designed to handle common document types like invoices, receipts, and more! Here\u2019s why you\u2019ll love using them:<\/p>\n\n- No Training Required:<\/strong> Prebuilt models<\/a> are ready to use out of the box & like having a team of document experts at your disposal. You don\u2019t need to spend time training them on your data.<\/span><\/li>\n
- High Accuracy:<\/strong> These models are built and maintained by Microsoft, ensuring high accuracy and reliability.<\/span><\/li>\n
- Easy Integration<\/strong>: With simple API calls, you can integrate these models into your applications quickly and efficiently.<\/span><\/li>\n
- Scalability:<\/strong> Azure\u2019s infrastructure ensures that your document processing can scale with your needs, handling everything from a few documents to millions.<\/span><\/li>\n<\/ul>\n
We define DocumentAnalysisClient\u00a0 to send requests to the Azure Document Intelligence<\/a>\u00a0service and receive responses containing the extracted text from the PDF resumes. It\u00a0provides operations for analyzing input documents using prebuilt and custom models through the\u00a0begin_analyze_document<\/code>\u00a0and\u00a0begin_analyze_document_from_url<\/code>\u00a0APIs.<\/p>\n<\/a><\/p>\nYou can also visualize how the document text extraction is performed using the\u00a0Document Intelligence Studio<\/a><\/p>\n<\/a><\/p>\nWhy are we chunking? <\/strong><\/h4>\nWhen dealing with content that exceeds the embedding limit of our model, we need to break it into smaller pieces. This is because the model that generates embeddings can only handle a certain amount of text at a time.<\/p>\n
Here, we divide the extracted text into fixed chunks of 500 tokens. These chunks are later passed to the text-embedding-small<\/strong> model to generate text embeddings, as the model has an input token limit of 8192<\/strong><\/p>\nThis process breaks down resumes into manageable pieces, making them easier to analyze and process. Think of it as turning a giant pizza into bite-sized slices\u2014much easier to handle!<\/p>\n
Note<\/strong>: There are different ways of chunking<\/a> content, and the method you choose can depend on your specific needs and the structure of the text you\u2019re working with.<\/p>\nWe now have our Resumes extracted & chunked:<\/p>\n
<\/a><\/p>\nStep 2: Generating Embeddings<\/strong><\/h2>\nNext, we generate embeddings from the chunked content using the Azure OpenAI API.<\/a> These embeddings are vector representations of the text, capturing the semantic meaning of the content.<\/p>\n<\/a><\/p>\nEach chunk now has a text embedding as seen below:<\/p>\n
<\/a><\/p>\nStep 3: Storing Embeddings in Azure SQL DB<\/strong><\/h3>\nWe then store these embeddings in Azure SQL DB using the new\u00a0Vector<\/a> data type. This allows us to perform similarity searches using built-in vector functions, enabling us to find the most suitable candidates based on their resumes.<\/p>\n<\/a><\/p>\nWe now have our data in the SQL DB which we can use for Vector Search. This means that even if the exact words don\u2019t match, the underlying meaning can still be understood and matched effectively.<\/p>\n
<\/a><\/p>\nFor example: If your search query is focused on project management skills, the embeddings can help find candidates with relevant experience, even if they use different terminology:<\/p>\n
Search Query: \u201cproject management\u201d<\/strong><\/p>\nMatched Candidate Skills:<\/strong><\/p>\n\n- \u201cLed multiple software development projects\u201d<\/li>\n
- \u201cCertified Scrum Master with experience in Agile methodologies\u201d<\/li>\n
- \u201cManaged cross-functional teams to deliver projects on time and within budget\u201d<\/li>\n<\/ul>\n
The embeddings capture the essence of \u201cproject management\u201d and match it with various related skills and experiences. It\u2019s like having a supercharged recruiter at your fingertips!<\/p>\n
Let’s now query our ResumeDocs table to get the top similar candidates given the User search query.<\/p>\n
Vector Search in SQL DB:<\/strong> Given any user search query, we can obtain the vector representation of that text. We then use this vector to calculate the cosine distance against all the resume embeddings stored in the database. By selecting only the closest matches, we can identify the resumes most relevant to the user\u2019s query. This helps in finding the most suitable candidates based on their resumes.<\/p>\nThe most common distance is the cosine similarity, which can be calculated quite easily in SQL with the help of the built-in vector distance functions.<\/p>\n
VECTOR_DISTANCE(‘distance metric’, V1, V2)<\/strong><\/p>\n<\/a><\/p>\nStep 4: Augmenting LLM Generation<\/strong><\/h3>\nFinally, we enhance language model generation with the Vector Search results from our Azure SQL Database. By informing a GPT-4o chat model with the results of our Vector search, we can provide rich, context-aware answers about candidates based on their resumes.<\/p>\n
The function\u00a0generate_completion<\/code> is defined to help ground the gpt-4o model with prompts and system instructions. Note that we are passing the results of the\u00a0vector_search_sql<\/code> we defined earlier<\/p>\n<\/a><\/p>\nWe can now create interactive loop where you can pose questions to the model and receive information grounded in your data.<\/p>\n
<\/a>\nLet’s look at Candidate 2:<\/p>\n<\/a>\n<\/a><\/p>\nWe also can get a short summary at the end along with a Fun Microsoft fact about the technology we are hiring for as we had defined in the prompt:<\/p>\n
<\/a><\/p>\nHence by utilizing Azure SQL DB\u2019s<\/strong> vector capabilities along with the power of Azure Document Intelligence and Azure OpenAI, you can totally transform your hiring game.<\/p>\nNo more missing out on the perfect candidate, and your interviews will be more on point than ever. But why stop at resumes? Dive into your own documents\u2014whether they\u2019re project reports, research papers\u2014and uncover hidden gems. This cool approach not only saves you time but also boosts the quality of your recruitment and document analysis. Happy exploring and hiring! \ud83d\ude80<\/p>\n
Private Preview Sign-up<\/h2>\n
We are currently accepting requests from customers who would like to participate in the private preview and try out the Native Vector Support for SQL feature. If you are interested, please fill out\u00a0https:\/\/aka.ms\/azuresql-vector-eap<\/a><\/p>\n
Getting Started<\/strong><\/h3>\nFor the purpose of this sample tutorial, we\u2019ll use a simple\u00a0Python notebook<\/a>\u00a0so that you can have everything up and running in less than 2 hours. Here\u2019s a quick overview of what you\u2019ll need to get started:<\/p>\nDataset<\/strong><\/h4>\nWe use a sample dataset from\u00a0Kaggle<\/a>\u00a0containing PDF resumes for this tutorial. For this tutorial, we will use 120 resumes from the\u00a0Information-Technology<\/strong>\u00a0folder<\/p>\nPrerequisites<\/strong><\/h4>\n\n- Azure Subscription<\/strong>:\u00a0Create one for free<\/a><\/span><\/li>\n
- Azure SQL Database<\/strong>:\u00a0Set up your database for free<\/a><\/span><\/li>\n
- Azure Document Intelligence<\/strong>\u00a0Create a Free Azure Document Intelligence resource<\/a><\/span><\/li>\n
- Azure OpenAI Access<\/strong>: Apply for access in the desired Azure subscription at\u00a0https:\/\/aka.ms\/oai\/access<\/a><\/span><\/li>\n
- Azure OpenAI Resource<\/strong>: Deploy an embeddings model (e.g.,\u00a0text-embedding-small\u00a0or\u00a0text-embedding-ada-002) and a\u00a0GPT-4.0\u00a0model for chat completion. Refer to the\u00a0resource deployment guide<\/a><\/span><\/li>\n
- Azure Data Studio<\/strong>: Download\u00a0here<\/a>\u00a0to manage your Azure SQL database and\u00a0execute the notebook<\/a><\/span><\/li>\n<\/ul>\n
Step 1: Extract & chunk content from PDF Resumes<\/strong><\/h3>\nFirst, we use Azure Document Intelligence to extract and chunk content from PDF resumes.<\/p>\n
Why Azure Document Intelligence?<\/strong><\/h4>\nAI Document Intelligence <\/a>is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately.\nThis means you can turn documents into usable data and shift your focus to acting on information rather than compiling it.<\/p>\nHere is an example of a Resume in PDF format from the dataset:\n<\/a><\/p>\nOne of the best parts about using Azure Document Intelligence is how easy it is to get started with prebuilt<\/strong> models. These models are designed to handle common document types like invoices, receipts, and more! Here\u2019s why you\u2019ll love using them:<\/p>\n\n- No Training Required:<\/strong> Prebuilt models<\/a> are ready to use out of the box & like having a team of document experts at your disposal. You don\u2019t need to spend time training them on your data.<\/span><\/li>\n
- High Accuracy:<\/strong> These models are built and maintained by Microsoft, ensuring high accuracy and reliability.<\/span><\/li>\n
- Easy Integration<\/strong>: With simple API calls, you can integrate these models into your applications quickly and efficiently.<\/span><\/li>\n
- Scalability:<\/strong> Azure\u2019s infrastructure ensures that your document processing can scale with your needs, handling everything from a few documents to millions.<\/span><\/li>\n<\/ul>\n
We define DocumentAnalysisClient\u00a0 to send requests to the Azure Document Intelligence<\/a>\u00a0service and receive responses containing the extracted text from the PDF resumes. It\u00a0provides operations for analyzing input documents using prebuilt and custom models through the\u00a0begin_analyze_document<\/code>\u00a0and\u00a0begin_analyze_document_from_url<\/code>\u00a0APIs.<\/p>\n<\/a><\/p>\nYou can also visualize how the document text extraction is performed using the\u00a0Document Intelligence Studio<\/a><\/p>\n<\/a><\/p>\nWhy are we chunking? <\/strong><\/h4>\nWhen dealing with content that exceeds the embedding limit of our model, we need to break it into smaller pieces. This is because the model that generates embeddings can only handle a certain amount of text at a time.<\/p>\n
Here, we divide the extracted text into fixed chunks of 500 tokens. These chunks are later passed to the text-embedding-small<\/strong> model to generate text embeddings, as the model has an input token limit of 8192<\/strong><\/p>\nThis process breaks down resumes into manageable pieces, making them easier to analyze and process. Think of it as turning a giant pizza into bite-sized slices\u2014much easier to handle!<\/p>\n
Note<\/strong>: There are different ways of chunking<\/a> content, and the method you choose can depend on your specific needs and the structure of the text you\u2019re working with.<\/p>\nWe now have our Resumes extracted & chunked:<\/p>\n
<\/a><\/p>\nStep 2: Generating Embeddings<\/strong><\/h2>\nNext, we generate embeddings from the chunked content using the Azure OpenAI API.<\/a> These embeddings are vector representations of the text, capturing the semantic meaning of the content.<\/p>\n<\/a><\/p>\nEach chunk now has a text embedding as seen below:<\/p>\n
<\/a><\/p>\nStep 3: Storing Embeddings in Azure SQL DB<\/strong><\/h3>\nWe then store these embeddings in Azure SQL DB using the new\u00a0Vector<\/a> data type. This allows us to perform similarity searches using built-in vector functions, enabling us to find the most suitable candidates based on their resumes.<\/p>\n<\/a><\/p>\nWe now have our data in the SQL DB which we can use for Vector Search. This means that even if the exact words don\u2019t match, the underlying meaning can still be understood and matched effectively.<\/p>\n
<\/a><\/p>\nFor example: If your search query is focused on project management skills, the embeddings can help find candidates with relevant experience, even if they use different terminology:<\/p>\n
Search Query: \u201cproject management\u201d<\/strong><\/p>\nMatched Candidate Skills:<\/strong><\/p>\n\n- \u201cLed multiple software development projects\u201d<\/li>\n
- \u201cCertified Scrum Master with experience in Agile methodologies\u201d<\/li>\n
- \u201cManaged cross-functional teams to deliver projects on time and within budget\u201d<\/li>\n<\/ul>\n
The embeddings capture the essence of \u201cproject management\u201d and match it with various related skills and experiences. It\u2019s like having a supercharged recruiter at your fingertips!<\/p>\n
Let’s now query our ResumeDocs table to get the top similar candidates given the User search query.<\/p>\n
Vector Search in SQL DB:<\/strong> Given any user search query, we can obtain the vector representation of that text. We then use this vector to calculate the cosine distance against all the resume embeddings stored in the database. By selecting only the closest matches, we can identify the resumes most relevant to the user\u2019s query. This helps in finding the most suitable candidates based on their resumes.<\/p>\nThe most common distance is the cosine similarity, which can be calculated quite easily in SQL with the help of the built-in vector distance functions.<\/p>\n
VECTOR_DISTANCE(‘distance metric’, V1, V2)<\/strong><\/p>\n<\/a><\/p>\nStep 4: Augmenting LLM Generation<\/strong><\/h3>\nFinally, we enhance language model generation with the Vector Search results from our Azure SQL Database. By informing a GPT-4o chat model with the results of our Vector search, we can provide rich, context-aware answers about candidates based on their resumes.<\/p>\n
The function\u00a0generate_completion<\/code> is defined to help ground the gpt-4o model with prompts and system instructions. Note that we are passing the results of the\u00a0vector_search_sql<\/code> we defined earlier<\/p>\n<\/a><\/p>\nWe can now create interactive loop where you can pose questions to the model and receive information grounded in your data.<\/p>\n
<\/a>\nLet’s look at Candidate 2:<\/p>\n<\/a>\n<\/a><\/p>\nWe also can get a short summary at the end along with a Fun Microsoft fact about the technology we are hiring for as we had defined in the prompt:<\/p>\n
<\/a><\/p>\nHence by utilizing Azure SQL DB\u2019s<\/strong> vector capabilities along with the power of Azure Document Intelligence and Azure OpenAI, you can totally transform your hiring game.<\/p>\nNo more missing out on the perfect candidate, and your interviews will be more on point than ever. But why stop at resumes? Dive into your own documents\u2014whether they\u2019re project reports, research papers\u2014and uncover hidden gems. This cool approach not only saves you time but also boosts the quality of your recruitment and document analysis. Happy exploring and hiring! \ud83d\ude80<\/p>\n
Private Preview Sign-up<\/h2>\n
We are currently accepting requests from customers who would like to participate in the private preview and try out the Native Vector Support for SQL feature. If you are interested, please fill out\u00a0https:\/\/aka.ms\/azuresql-vector-eap<\/a><\/p>\n
Dataset<\/strong><\/h4>\nWe use a sample dataset from\u00a0Kaggle<\/a>\u00a0containing PDF resumes for this tutorial. For this tutorial, we will use 120 resumes from the\u00a0Information-Technology<\/strong>\u00a0folder<\/p>\nPrerequisites<\/strong><\/h4>\n\n- Azure Subscription<\/strong>:\u00a0Create one for free<\/a><\/span><\/li>\n
- Azure SQL Database<\/strong>:\u00a0Set up your database for free<\/a><\/span><\/li>\n
- Azure Document Intelligence<\/strong>\u00a0Create a Free Azure Document Intelligence resource<\/a><\/span><\/li>\n
- Azure OpenAI Access<\/strong>: Apply for access in the desired Azure subscription at\u00a0https:\/\/aka.ms\/oai\/access<\/a><\/span><\/li>\n
- Azure OpenAI Resource<\/strong>: Deploy an embeddings model (e.g.,\u00a0text-embedding-small\u00a0or\u00a0text-embedding-ada-002) and a\u00a0GPT-4.0\u00a0model for chat completion. Refer to the\u00a0resource deployment guide<\/a><\/span><\/li>\n
- Azure Data Studio<\/strong>: Download\u00a0here<\/a>\u00a0to manage your Azure SQL database and\u00a0execute the notebook<\/a><\/span><\/li>\n<\/ul>\n
Step 1: Extract & chunk content from PDF Resumes<\/strong><\/h3>\nFirst, we use Azure Document Intelligence to extract and chunk content from PDF resumes.<\/p>\n
Why Azure Document Intelligence?<\/strong><\/h4>\nAI Document Intelligence <\/a>is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately.\nThis means you can turn documents into usable data and shift your focus to acting on information rather than compiling it.<\/p>\nHere is an example of a Resume in PDF format from the dataset:\n<\/a><\/p>\nOne of the best parts about using Azure Document Intelligence is how easy it is to get started with prebuilt<\/strong> models. These models are designed to handle common document types like invoices, receipts, and more! Here\u2019s why you\u2019ll love using them:<\/p>\n\n- No Training Required:<\/strong> Prebuilt models<\/a> are ready to use out of the box & like having a team of document experts at your disposal. You don\u2019t need to spend time training them on your data.<\/span><\/li>\n
- High Accuracy:<\/strong> These models are built and maintained by Microsoft, ensuring high accuracy and reliability.<\/span><\/li>\n
- Easy Integration<\/strong>: With simple API calls, you can integrate these models into your applications quickly and efficiently.<\/span><\/li>\n
- Scalability:<\/strong> Azure\u2019s infrastructure ensures that your document processing can scale with your needs, handling everything from a few documents to millions.<\/span><\/li>\n<\/ul>\n
We define DocumentAnalysisClient\u00a0 to send requests to the Azure Document Intelligence<\/a>\u00a0service and receive responses containing the extracted text from the PDF resumes. It\u00a0provides operations for analyzing input documents using prebuilt and custom models through the\u00a0begin_analyze_document<\/code>\u00a0and\u00a0begin_analyze_document_from_url<\/code>\u00a0APIs.<\/p>\n<\/a><\/p>\nYou can also visualize how the document text extraction is performed using the\u00a0Document Intelligence Studio<\/a><\/p>\n<\/a><\/p>\nWhy are we chunking? <\/strong><\/h4>\nWhen dealing with content that exceeds the embedding limit of our model, we need to break it into smaller pieces. This is because the model that generates embeddings can only handle a certain amount of text at a time.<\/p>\n
Here, we divide the extracted text into fixed chunks of 500 tokens. These chunks are later passed to the text-embedding-small<\/strong> model to generate text embeddings, as the model has an input token limit of 8192<\/strong><\/p>\nThis process breaks down resumes into manageable pieces, making them easier to analyze and process. Think of it as turning a giant pizza into bite-sized slices\u2014much easier to handle!<\/p>\n
Note<\/strong>: There are different ways of chunking<\/a> content, and the method you choose can depend on your specific needs and the structure of the text you\u2019re working with.<\/p>\nWe now have our Resumes extracted & chunked:<\/p>\n
<\/a><\/p>\nStep 2: Generating Embeddings<\/strong><\/h2>\nNext, we generate embeddings from the chunked content using the Azure OpenAI API.<\/a> These embeddings are vector representations of the text, capturing the semantic meaning of the content.<\/p>\n<\/a><\/p>\nEach chunk now has a text embedding as seen below:<\/p>\n
<\/a><\/p>\nStep 3: Storing Embeddings in Azure SQL DB<\/strong><\/h3>\nWe then store these embeddings in Azure SQL DB using the new\u00a0Vector<\/a> data type. This allows us to perform similarity searches using built-in vector functions, enabling us to find the most suitable candidates based on their resumes.<\/p>\n<\/a><\/p>\nWe now have our data in the SQL DB which we can use for Vector Search. This means that even if the exact words don\u2019t match, the underlying meaning can still be understood and matched effectively.<\/p>\n
<\/a><\/p>\nFor example: If your search query is focused on project management skills, the embeddings can help find candidates with relevant experience, even if they use different terminology:<\/p>\n
Search Query: \u201cproject management\u201d<\/strong><\/p>\nMatched Candidate Skills:<\/strong><\/p>\n\n- \u201cLed multiple software development projects\u201d<\/li>\n
- \u201cCertified Scrum Master with experience in Agile methodologies\u201d<\/li>\n
- \u201cManaged cross-functional teams to deliver projects on time and within budget\u201d<\/li>\n<\/ul>\n
The embeddings capture the essence of \u201cproject management\u201d and match it with various related skills and experiences. It\u2019s like having a supercharged recruiter at your fingertips!<\/p>\n
Let’s now query our ResumeDocs table to get the top similar candidates given the User search query.<\/p>\n
Vector Search in SQL DB:<\/strong> Given any user search query, we can obtain the vector representation of that text. We then use this vector to calculate the cosine distance against all the resume embeddings stored in the database. By selecting only the closest matches, we can identify the resumes most relevant to the user\u2019s query. This helps in finding the most suitable candidates based on their resumes.<\/p>\nThe most common distance is the cosine similarity, which can be calculated quite easily in SQL with the help of the built-in vector distance functions.<\/p>\n
VECTOR_DISTANCE(‘distance metric’, V1, V2)<\/strong><\/p>\n<\/a><\/p>\nStep 4: Augmenting LLM Generation<\/strong><\/h3>\nFinally, we enhance language model generation with the Vector Search results from our Azure SQL Database. By informing a GPT-4o chat model with the results of our Vector search, we can provide rich, context-aware answers about candidates based on their resumes.<\/p>\n
The function\u00a0generate_completion<\/code> is defined to help ground the gpt-4o model with prompts and system instructions. Note that we are passing the results of the\u00a0vector_search_sql<\/code> we defined earlier<\/p>\n<\/a><\/p>\nWe can now create interactive loop where you can pose questions to the model and receive information grounded in your data.<\/p>\n
<\/a>\nLet’s look at Candidate 2:<\/p>\n<\/a>\n<\/a><\/p>\nWe also can get a short summary at the end along with a Fun Microsoft fact about the technology we are hiring for as we had defined in the prompt:<\/p>\n
<\/a><\/p>\nHence by utilizing Azure SQL DB\u2019s<\/strong> vector capabilities along with the power of Azure Document Intelligence and Azure OpenAI, you can totally transform your hiring game.<\/p>\nNo more missing out on the perfect candidate, and your interviews will be more on point than ever. But why stop at resumes? Dive into your own documents\u2014whether they\u2019re project reports, research papers\u2014and uncover hidden gems. This cool approach not only saves you time but also boosts the quality of your recruitment and document analysis. Happy exploring and hiring! \ud83d\ude80<\/p>\n
Private Preview Sign-up<\/h2>\n
We are currently accepting requests from customers who would like to participate in the private preview and try out the Native Vector Support for SQL feature. If you are interested, please fill out\u00a0https:\/\/aka.ms\/azuresql-vector-eap<\/a><\/p>\n
Prerequisites<\/strong><\/h4>\n\n- Azure Subscription<\/strong>:\u00a0Create one for free<\/a><\/span><\/li>\n
- Azure SQL Database<\/strong>:\u00a0Set up your database for free<\/a><\/span><\/li>\n
- Azure Document Intelligence<\/strong>\u00a0Create a Free Azure Document Intelligence resource<\/a><\/span><\/li>\n
- Azure OpenAI Access<\/strong>: Apply for access in the desired Azure subscription at\u00a0https:\/\/aka.ms\/oai\/access<\/a><\/span><\/li>\n
- Azure OpenAI Resource<\/strong>: Deploy an embeddings model (e.g.,\u00a0text-embedding-small\u00a0or\u00a0text-embedding-ada-002) and a\u00a0GPT-4.0\u00a0model for chat completion. Refer to the\u00a0resource deployment guide<\/a><\/span><\/li>\n
- Azure Data Studio<\/strong>: Download\u00a0here<\/a>\u00a0to manage your Azure SQL database and\u00a0execute the notebook<\/a><\/span><\/li>\n<\/ul>\n
Step 1: Extract & chunk content from PDF Resumes<\/strong><\/h3>\nFirst, we use Azure Document Intelligence to extract and chunk content from PDF resumes.<\/p>\n
Why Azure Document Intelligence?<\/strong><\/h4>\nAI Document Intelligence <\/a>is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately.\nThis means you can turn documents into usable data and shift your focus to acting on information rather than compiling it.<\/p>\nHere is an example of a Resume in PDF format from the dataset:\n<\/a><\/p>\nOne of the best parts about using Azure Document Intelligence is how easy it is to get started with prebuilt<\/strong> models. These models are designed to handle common document types like invoices, receipts, and more! Here\u2019s why you\u2019ll love using them:<\/p>\n\n- No Training Required:<\/strong> Prebuilt models<\/a> are ready to use out of the box & like having a team of document experts at your disposal. You don\u2019t need to spend time training them on your data.<\/span><\/li>\n
- High Accuracy:<\/strong> These models are built and maintained by Microsoft, ensuring high accuracy and reliability.<\/span><\/li>\n
- Easy Integration<\/strong>: With simple API calls, you can integrate these models into your applications quickly and efficiently.<\/span><\/li>\n
- Scalability:<\/strong> Azure\u2019s infrastructure ensures that your document processing can scale with your needs, handling everything from a few documents to millions.<\/span><\/li>\n<\/ul>\n
We define DocumentAnalysisClient\u00a0 to send requests to the Azure Document Intelligence<\/a>\u00a0service and receive responses containing the extracted text from the PDF resumes. It\u00a0provides operations for analyzing input documents using prebuilt and custom models through the\u00a0begin_analyze_document<\/code>\u00a0and\u00a0begin_analyze_document_from_url<\/code>\u00a0APIs.<\/p>\n<\/a><\/p>\nYou can also visualize how the document text extraction is performed using the\u00a0Document Intelligence Studio<\/a><\/p>\n<\/a><\/p>\nWhy are we chunking? <\/strong><\/h4>\nWhen dealing with content that exceeds the embedding limit of our model, we need to break it into smaller pieces. This is because the model that generates embeddings can only handle a certain amount of text at a time.<\/p>\n
Here, we divide the extracted text into fixed chunks of 500 tokens. These chunks are later passed to the text-embedding-small<\/strong> model to generate text embeddings, as the model has an input token limit of 8192<\/strong><\/p>\nThis process breaks down resumes into manageable pieces, making them easier to analyze and process. Think of it as turning a giant pizza into bite-sized slices\u2014much easier to handle!<\/p>\n
Note<\/strong>: There are different ways of chunking<\/a> content, and the method you choose can depend on your specific needs and the structure of the text you\u2019re working with.<\/p>\nWe now have our Resumes extracted & chunked:<\/p>\n
<\/a><\/p>\nStep 2: Generating Embeddings<\/strong><\/h2>\nNext, we generate embeddings from the chunked content using the Azure OpenAI API.<\/a> These embeddings are vector representations of the text, capturing the semantic meaning of the content.<\/p>\n<\/a><\/p>\nEach chunk now has a text embedding as seen below:<\/p>\n
<\/a><\/p>\nStep 3: Storing Embeddings in Azure SQL DB<\/strong><\/h3>\nWe then store these embeddings in Azure SQL DB using the new\u00a0Vector<\/a> data type. This allows us to perform similarity searches using built-in vector functions, enabling us to find the most suitable candidates based on their resumes.<\/p>\n<\/a><\/p>\nWe now have our data in the SQL DB which we can use for Vector Search. This means that even if the exact words don\u2019t match, the underlying meaning can still be understood and matched effectively.<\/p>\n
<\/a><\/p>\nFor example: If your search query is focused on project management skills, the embeddings can help find candidates with relevant experience, even if they use different terminology:<\/p>\n
Search Query: \u201cproject management\u201d<\/strong><\/p>\nMatched Candidate Skills:<\/strong><\/p>\n\n- \u201cLed multiple software development projects\u201d<\/li>\n
- \u201cCertified Scrum Master with experience in Agile methodologies\u201d<\/li>\n
- \u201cManaged cross-functional teams to deliver projects on time and within budget\u201d<\/li>\n<\/ul>\n
The embeddings capture the essence of \u201cproject management\u201d and match it with various related skills and experiences. It\u2019s like having a supercharged recruiter at your fingertips!<\/p>\n
Let’s now query our ResumeDocs table to get the top similar candidates given the User search query.<\/p>\n
Vector Search in SQL DB:<\/strong> Given any user search query, we can obtain the vector representation of that text. We then use this vector to calculate the cosine distance against all the resume embeddings stored in the database. By selecting only the closest matches, we can identify the resumes most relevant to the user\u2019s query. This helps in finding the most suitable candidates based on their resumes.<\/p>\nThe most common distance is the cosine similarity, which can be calculated quite easily in SQL with the help of the built-in vector distance functions.<\/p>\n
VECTOR_DISTANCE(‘distance metric’, V1, V2)<\/strong><\/p>\n<\/a><\/p>\nStep 4: Augmenting LLM Generation<\/strong><\/h3>\nFinally, we enhance language model generation with the Vector Search results from our Azure SQL Database. By informing a GPT-4o chat model with the results of our Vector search, we can provide rich, context-aware answers about candidates based on their resumes.<\/p>\n
The function\u00a0generate_completion<\/code> is defined to help ground the gpt-4o model with prompts and system instructions. Note that we are passing the results of the\u00a0vector_search_sql<\/code> we defined earlier<\/p>\n<\/a><\/p>\nWe can now create interactive loop where you can pose questions to the model and receive information grounded in your data.<\/p>\n
<\/a>\nLet’s look at Candidate 2:<\/p>\n<\/a>\n<\/a><\/p>\nWe also can get a short summary at the end along with a Fun Microsoft fact about the technology we are hiring for as we had defined in the prompt:<\/p>\n
<\/a><\/p>\nHence by utilizing Azure SQL DB\u2019s<\/strong> vector capabilities along with the power of Azure Document Intelligence and Azure OpenAI, you can totally transform your hiring game.<\/p>\nNo more missing out on the perfect candidate, and your interviews will be more on point than ever. But why stop at resumes? Dive into your own documents\u2014whether they\u2019re project reports, research papers\u2014and uncover hidden gems. This cool approach not only saves you time but also boosts the quality of your recruitment and document analysis. Happy exploring and hiring! \ud83d\ude80<\/p>\n
Private Preview Sign-up<\/h2>\n
We are currently accepting requests from customers who would like to participate in the private preview and try out the Native Vector Support for SQL feature. If you are interested, please fill out\u00a0https:\/\/aka.ms\/azuresql-vector-eap<\/a><\/p>\n
Step 1: Extract & chunk content from PDF Resumes<\/strong><\/h3>\nFirst, we use Azure Document Intelligence to extract and chunk content from PDF resumes.<\/p>\n
Why Azure Document Intelligence?<\/strong><\/h4>\nAI Document Intelligence <\/a>is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately.\nThis means you can turn documents into usable data and shift your focus to acting on information rather than compiling it.<\/p>\nHere is an example of a Resume in PDF format from the dataset:\n<\/a><\/p>\nOne of the best parts about using Azure Document Intelligence is how easy it is to get started with prebuilt<\/strong> models. These models are designed to handle common document types like invoices, receipts, and more! Here\u2019s why you\u2019ll love using them:<\/p>\n\n- No Training Required:<\/strong> Prebuilt models<\/a> are ready to use out of the box & like having a team of document experts at your disposal. You don\u2019t need to spend time training them on your data.<\/span><\/li>\n
- High Accuracy:<\/strong> These models are built and maintained by Microsoft, ensuring high accuracy and reliability.<\/span><\/li>\n
- Easy Integration<\/strong>: With simple API calls, you can integrate these models into your applications quickly and efficiently.<\/span><\/li>\n
- Scalability:<\/strong> Azure\u2019s infrastructure ensures that your document processing can scale with your needs, handling everything from a few documents to millions.<\/span><\/li>\n<\/ul>\n
We define DocumentAnalysisClient\u00a0 to send requests to the Azure Document Intelligence<\/a>\u00a0service and receive responses containing the extracted text from the PDF resumes. It\u00a0provides operations for analyzing input documents using prebuilt and custom models through the\u00a0begin_analyze_document<\/code>\u00a0and\u00a0begin_analyze_document_from_url<\/code>\u00a0APIs.<\/p>\n<\/a><\/p>\nYou can also visualize how the document text extraction is performed using the\u00a0Document Intelligence Studio<\/a><\/p>\n<\/a><\/p>\nWhy are we chunking? <\/strong><\/h4>\nWhen dealing with content that exceeds the embedding limit of our model, we need to break it into smaller pieces. This is because the model that generates embeddings can only handle a certain amount of text at a time.<\/p>\n
Here, we divide the extracted text into fixed chunks of 500 tokens. These chunks are later passed to the text-embedding-small<\/strong> model to generate text embeddings, as the model has an input token limit of 8192<\/strong><\/p>\nThis process breaks down resumes into manageable pieces, making them easier to analyze and process. Think of it as turning a giant pizza into bite-sized slices\u2014much easier to handle!<\/p>\n
Note<\/strong>: There are different ways of chunking<\/a> content, and the method you choose can depend on your specific needs and the structure of the text you\u2019re working with.<\/p>\nWe now have our Resumes extracted & chunked:<\/p>\n
<\/a><\/p>\nStep 2: Generating Embeddings<\/strong><\/h2>\nNext, we generate embeddings from the chunked content using the Azure OpenAI API.<\/a> These embeddings are vector representations of the text, capturing the semantic meaning of the content.<\/p>\n<\/a><\/p>\nEach chunk now has a text embedding as seen below:<\/p>\n
<\/a><\/p>\nStep 3: Storing Embeddings in Azure SQL DB<\/strong><\/h3>\nWe then store these embeddings in Azure SQL DB using the new\u00a0Vector<\/a> data type. This allows us to perform similarity searches using built-in vector functions, enabling us to find the most suitable candidates based on their resumes.<\/p>\n<\/a><\/p>\nWe now have our data in the SQL DB which we can use for Vector Search. This means that even if the exact words don\u2019t match, the underlying meaning can still be understood and matched effectively.<\/p>\n
<\/a><\/p>\nFor example: If your search query is focused on project management skills, the embeddings can help find candidates with relevant experience, even if they use different terminology:<\/p>\n
Search Query: \u201cproject management\u201d<\/strong><\/p>\nMatched Candidate Skills:<\/strong><\/p>\n\n- \u201cLed multiple software development projects\u201d<\/li>\n
- \u201cCertified Scrum Master with experience in Agile methodologies\u201d<\/li>\n
- \u201cManaged cross-functional teams to deliver projects on time and within budget\u201d<\/li>\n<\/ul>\n
The embeddings capture the essence of \u201cproject management\u201d and match it with various related skills and experiences. It\u2019s like having a supercharged recruiter at your fingertips!<\/p>\n
Let’s now query our ResumeDocs table to get the top similar candidates given the User search query.<\/p>\n
Vector Search in SQL DB:<\/strong> Given any user search query, we can obtain the vector representation of that text. We then use this vector to calculate the cosine distance against all the resume embeddings stored in the database. By selecting only the closest matches, we can identify the resumes most relevant to the user\u2019s query. This helps in finding the most suitable candidates based on their resumes.<\/p>\nThe most common distance is the cosine similarity, which can be calculated quite easily in SQL with the help of the built-in vector distance functions.<\/p>\n
VECTOR_DISTANCE(‘distance metric’, V1, V2)<\/strong><\/p>\n<\/a><\/p>\nStep 4: Augmenting LLM Generation<\/strong><\/h3>\nFinally, we enhance language model generation with the Vector Search results from our Azure SQL Database. By informing a GPT-4o chat model with the results of our Vector search, we can provide rich, context-aware answers about candidates based on their resumes.<\/p>\n
The function\u00a0generate_completion<\/code> is defined to help ground the gpt-4o model with prompts and system instructions. Note that we are passing the results of the\u00a0vector_search_sql<\/code> we defined earlier<\/p>\n<\/a><\/p>\nWe can now create interactive loop where you can pose questions to the model and receive information grounded in your data.<\/p>\n
<\/a>\nLet’s look at Candidate 2:<\/p>\n<\/a>\n<\/a><\/p>\nWe also can get a short summary at the end along with a Fun Microsoft fact about the technology we are hiring for as we had defined in the prompt:<\/p>\n
<\/a><\/p>\nHence by utilizing Azure SQL DB\u2019s<\/strong> vector capabilities along with the power of Azure Document Intelligence and Azure OpenAI, you can totally transform your hiring game.<\/p>\nNo more missing out on the perfect candidate, and your interviews will be more on point than ever. But why stop at resumes? Dive into your own documents\u2014whether they\u2019re project reports, research papers\u2014and uncover hidden gems. This cool approach not only saves you time but also boosts the quality of your recruitment and document analysis. Happy exploring and hiring! \ud83d\ude80<\/p>\n
Private Preview Sign-up<\/h2>\n
We are currently accepting requests from customers who would like to participate in the private preview and try out the Native Vector Support for SQL feature. If you are interested, please fill out\u00a0https:\/\/aka.ms\/azuresql-vector-eap<\/a><\/p>\n
AI Document Intelligence <\/a>is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately.\nThis means you can turn documents into usable data and shift your focus to acting on information rather than compiling it.<\/p>\n Here is an example of a Resume in PDF format from the dataset:\n One of the best parts about using Azure Document Intelligence is how easy it is to get started with prebuilt<\/strong> models. These models are designed to handle common document types like invoices, receipts, and more! Here\u2019s why you\u2019ll love using them:<\/p>\n We define DocumentAnalysisClient\u00a0 to send requests to the Azure Document Intelligence<\/a>\u00a0service and receive responses containing the extracted text from the PDF resumes. It\u00a0provides operations for analyzing input documents using prebuilt and custom models through the\u00a0 You can also visualize how the document text extraction is performed using the\u00a0Document Intelligence Studio<\/a><\/p>\n When dealing with content that exceeds the embedding limit of our model, we need to break it into smaller pieces. This is because the model that generates embeddings can only handle a certain amount of text at a time.<\/p>\n Here, we divide the extracted text into fixed chunks of 500 tokens. These chunks are later passed to the text-embedding-small<\/strong> model to generate text embeddings, as the model has an input token limit of 8192<\/strong><\/p>\n This process breaks down resumes into manageable pieces, making them easier to analyze and process. Think of it as turning a giant pizza into bite-sized slices\u2014much easier to handle!<\/p>\n Note<\/strong>: There are different ways of chunking<\/a> content, and the method you choose can depend on your specific needs and the structure of the text you\u2019re working with.<\/p>\n We now have our Resumes extracted & chunked:<\/p>\n Next, we generate embeddings from the chunked content using the Azure OpenAI API.<\/a> These embeddings are vector representations of the text, capturing the semantic meaning of the content.<\/p>\n Each chunk now has a text embedding as seen below:<\/p>\n We then store these embeddings in Azure SQL DB using the new\u00a0Vector<\/a> data type. This allows us to perform similarity searches using built-in vector functions, enabling us to find the most suitable candidates based on their resumes.<\/p>\n We now have our data in the SQL DB which we can use for Vector Search. This means that even if the exact words don\u2019t match, the underlying meaning can still be understood and matched effectively.<\/p>\n For example: If your search query is focused on project management skills, the embeddings can help find candidates with relevant experience, even if they use different terminology:<\/p>\n Search Query: \u201cproject management\u201d<\/strong><\/p>\n Matched Candidate Skills:<\/strong><\/p>\n The embeddings capture the essence of \u201cproject management\u201d and match it with various related skills and experiences. It\u2019s like having a supercharged recruiter at your fingertips!<\/p>\n Let’s now query our ResumeDocs table to get the top similar candidates given the User search query.<\/p>\n Vector Search in SQL DB:<\/strong> Given any user search query, we can obtain the vector representation of that text. We then use this vector to calculate the cosine distance against all the resume embeddings stored in the database. By selecting only the closest matches, we can identify the resumes most relevant to the user\u2019s query. This helps in finding the most suitable candidates based on their resumes.<\/p>\n The most common distance is the cosine similarity, which can be calculated quite easily in SQL with the help of the built-in vector distance functions.<\/p>\n VECTOR_DISTANCE(‘distance metric’, V1, V2)<\/strong><\/p>\n Finally, we enhance language model generation with the Vector Search results from our Azure SQL Database. By informing a GPT-4o chat model with the results of our Vector search, we can provide rich, context-aware answers about candidates based on their resumes.<\/p>\n The function\u00a0 We can now create interactive loop where you can pose questions to the model and receive information grounded in your data.<\/p>\n We also can get a short summary at the end along with a Fun Microsoft fact about the technology we are hiring for as we had defined in the prompt:<\/p>\n Hence by utilizing Azure SQL DB\u2019s<\/strong> vector capabilities along with the power of Azure Document Intelligence and Azure OpenAI, you can totally transform your hiring game.<\/p>\n No more missing out on the perfect candidate, and your interviews will be more on point than ever. But why stop at resumes? Dive into your own documents\u2014whether they\u2019re project reports, research papers\u2014and uncover hidden gems. This cool approach not only saves you time but also boosts the quality of your recruitment and document analysis. Happy exploring and hiring! \ud83d\ude80<\/p>\n We are currently accepting requests from customers who would like to participate in the private preview and try out the Native Vector Support for SQL feature. If you are interested, please fill out\u00a0https:\/\/aka.ms\/azuresql-vector-eap<\/a><\/p>\n<\/a><\/p>\n\n
begin_analyze_document<\/code>\u00a0and\u00a0begin_analyze_document_from_url<\/code>\u00a0APIs.<\/p>\n<\/a><\/p>\n<\/a><\/p>\nWhy are we chunking? <\/strong><\/h4>\n
<\/a><\/p>\nStep 2: Generating Embeddings<\/strong><\/h2>\n
<\/a><\/p>\n<\/a><\/p>\nStep 3: Storing Embeddings in Azure SQL DB<\/strong><\/h3>\n
<\/a><\/p>\n<\/a><\/p>\n\n
<\/a><\/p>\nStep 4: Augmenting LLM Generation<\/strong><\/h3>\n
generate_completion<\/code> is defined to help ground the gpt-4o model with prompts and system instructions. Note that we are passing the results of the\u00a0vector_search_sql<\/code> we defined earlier<\/p>\n<\/a><\/p>\n<\/a>\nLet’s look at Candidate 2:<\/p>\n<\/a>\n<\/a><\/p>\n<\/a><\/p>\nPrivate Preview Sign-up<\/h2>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2024\/05\/QRCode-for-Vector-Support-in-Azure-SQL-Database-Private-Preview-Sign-up-1-300x300.png\")
<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"