{"id":42295,"date":"2024-09-20T00:00:12","date_gmt":"2024-09-20T07:00:12","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/premier-developer\/?p=42295"},"modified":"2024-09-17T06:05:07","modified_gmt":"2024-09-17T13:05:07","slug":"autogen-rag","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/premier-developer\/autogen-rag\/","title":{"rendered":"Autogen RAG"},"content":{"rendered":"<p><a href=\"https:\/\/www.linkedin.com\/in\/linkai-yu-75111417\/\" target=\"_blank\" rel=\"noopener noreferrer\">Linkai Yu<\/a> explores the retrieval-augemented generation pattern with Azure OpenAI Assistants API.<\/p>\n<hr \/>\n<p>The Retrieval-Augmented Generation (RAG) pattern is the standard for integrating ground AI with local data. One powerful way to implement RAG is through the function call or tool call feature in the Function Call API. When examining the samples provided by the Azure OpenAI Assistant API Function Call, <a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/ai-services\/openai\/how-to\/function-calling#parallel-function-calling-with-multiple-functions\">Completion API Function Call<\/a>, or <a href=\"https:\/\/microsoft.github.io\/autogen\/docs\/notebooks\/agentchat_function_call_currency_calculator\">Autogen Function Call<\/a>, they all require the function metadata to be specified in the code. Imagine you have hundreds of data I\/O functions, ranging from SQL queries to microservice calls, and your application needs to support various user prompts that cannot be reliably predicted by pre-defining the functions to call.<\/p>\n<p>The question then becomes, &#8220;Is there is a way to figure out the functions needed after the user prompt is received, and then configure the Tools parameter (in the Completion API and Assistant API) to configure Autogen\u2019s agent llm_config and function map dynamically?&#8221;<\/p>\n<p>This blog tries to answer this question. We are going to use Autogen as an example since it supports function call very well.<\/p>\n<p>The benefit of this approach is that once you have this architecture in place, you can connect up with backend data IO functions. This will result in an application that allows user to ask variety of questions in different combinations without developer having to change the app logic code. Developers can add functions to the list without having to change the application file.<\/p>\n<p>Let\u2019s suppose your app has a list of functions:<\/p>\n<ul>\n<li>read_file<\/li>\n<li>save_to_file<\/li>\n<li>get_health_insurance_account<\/li>\n<li>get_health_insurance_policy<\/li>\n<li>get_policy_benefits<\/li>\n<li>summarize_policy_content<\/li>\n<li>find_careproviders<\/li>\n<li>analyze_sentiment<\/li>\n<li>ask_a_question<\/li>\n<\/ul>\n<p><strong>A user may ask this question:<\/strong><\/p>\n<p>\u201cI have an insurance account with me (Brad Smith) as policy holder. I\u2019d like to get the benefits, have it summarized in a paragraph and save it to file c:\\mybenefits.txt\u201d<\/p>\n<p><strong>Or this question:<\/strong><\/p>\n<p>\u201cI have an insurance account with me (Brad Smith) as policy holder. I\u2019d like to find healthcare providers in my area (New York), and save it to file c:\\careproviders.txt\u201d<\/p>\n<p><strong>Or&#8230; other questions so long as your app have data IO functions to support data operation.<\/strong><\/p>\n<p>The algorithm to approach this use case is this:<\/p>\n<ol>\n<li>We provide (register) a generic function that knows how to find the function name given a description of a function. E.g. RegisterFunction(\u201cfind insurance policy\u201d)<\/li>\n<li>We ask AI to make a function call to RegisterFunction(&lt;description&gt;) if it needs information from the user agent.<\/li>\n<li>The user agent invokes RegisterFunction() with the description it received from AI<\/li>\n<li>The RegisterFunction() does a vector DB search for function name using the function description.<\/li>\n<li>With the function name, it looks up the actual function in the functions dictionary (key=name, value=function). With the function, it registers it with user agent (for function map) and assistant agent (for llm_config). The function information is made available to AI to call.<\/li>\n<li>AI makes the right function call. User agent invokes the function.<\/li>\n<\/ol>\n<p><strong>To make this process happen, we need these data structures:<\/strong><\/p>\n<ul>\n<li>A dictionary of all the functions.\n<ul>\n<li>We use this dictionary to find the function object so that we can register it with user agent and assistant agent<\/li>\n<\/ul>\n<\/li>\n<li>A vector DB that has three fields: id, name, description.\n<ul>\n<li>The id is to maintain record uniqueness. The name is for the function name. The description is a text description of the function for vector search.<\/li>\n<\/ul>\n<\/li>\n<li>A decorator to decorate the function with a description property.\n<ul>\n<li>This allows the vector DB to have the function description field. It also allows RegisterFunction() function to provide the user agent the function description.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>This is what the function table used to populate vector DB:<\/strong><\/p>\n<p>functions_table = \u00a0[<\/p>\n<p>{&#8220;id&#8221;: &#8220;1&#8221;,&#8221;func&#8221;: read_file},<\/p>\n<p>{&#8220;id&#8221;: &#8220;2&#8221;,&#8221;func&#8221;: save_to_file},<\/p>\n<p>{&#8220;id&#8221;: &#8220;3&#8221;,&#8221;func&#8221;: get_health_insurance_account},<\/p>\n<p>{&#8220;id&#8221;: &#8220;4&#8221;,&#8221;func&#8221;: get_health_insurance_policy },<\/p>\n<p>{&#8220;id&#8221;: &#8220;5&#8221;,&#8221;func&#8221;: get_policy_benefits },<\/p>\n<p>{&#8220;id&#8221;: &#8220;6&#8221;,&#8221;func&#8221;: summarize_policy_content },<\/p>\n<p>{&#8220;id&#8221;: &#8220;7&#8221;,&#8221;func&#8221;: find_careproviders },<\/p>\n<p>{&#8220;id&#8221;: &#8220;8&#8221;,&#8221;func&#8221;: analyze_sentiment },<\/p>\n<p>{&#8220;id&#8221;: &#8220;9&#8221;,&#8221;func&#8221;: ask_a_question},<\/p>\n<p>]<\/p>\n<p><strong>This is the function dictionary created at runtime using the functions_table:<\/strong><\/p>\n<p>functions_dict = {item[&#8220;func&#8221;].__name__: item[&#8220;func&#8221;] for item in functions_table}<\/p>\n<p>This is the function decorator for description<\/p>\n<p>def desc(desc):<\/p>\n<p>def wrapper(f):<\/p>\n<p>f.__desc__ = desc<\/p>\n<p>return f<\/p>\n<p>return wrapper<\/p>\n<p><strong>Example usage of the decorator:<\/strong><\/p>\n<p>@desc(\u201cread a file\u201d)<\/p>\n<p>def read_file(file_path: Annotated[str, &#8220;Name and path of file to read.&#8221;]) -&gt; Annotated[str, &#8220;file content&#8221;]:<\/p>\n<p><strong>Code explanation:<\/strong><\/p>\n<p>We put all the functions in a separate file to be included by the main file. This will allow the update of the functions file without having to update the main file.<\/p>\n<ol>\n<li><strong>Define the data IO functions:<\/strong>\n<pre class=\"prettyprint language-py\"><code class=\"language-py\">def desc(desc):\r\n\u00a0 \u00a0 def wrapper(f):\r\n\u00a0 \u00a0 \u00a0 \u00a0 f.__desc__ = desc\r\n\u00a0 \u00a0 \u00a0 \u00a0 return f\r\n\u00a0 \u00a0 return wrapper\r\n# define custom functions:\r\n@desc(\"read the content of a file\")\r\ndef read_file(file_path: Annotated[str, \"Name and path of file to read.\"]) -&gt; Annotated[str, \"file content\"]:\r\n\u2026\r\n@desc(\"save the content to a file\")\r\ndef save_to_file(file_path: Annotated[str, \"full path to the file\"], content: Annotated[str, \"content\"]) -&gt; Annotated[str, \"status: success or error\"]:<\/code><\/pre>\n<\/li>\n<\/ol>\n<p><strong>2. Enter all the functions in a function_table:<\/strong><\/p>\n<pre class=\"prettyprint language-py\"><code class=\"language-py\">functions_table = \u00a0[\r\n\u00a0 \u00a0 {\"id\": \"1\",\"func\": read_file},\r\n\u00a0 \u00a0 {\"id\": \"2\",\"func\": save_to_file},\r\n\u00a0 \u00a0 {\"id\": \"3\",\"func\": get_health_insurance_account},\r\n\u00a0 \u00a0 {\"id\": \"4\",\"func\": get_health_insurance_policy },\r\n\u00a0 \u00a0 {\"id\": \"5\",\"func\": get_policy_benefits },\r\n\u00a0 \u00a0 {\"id\": \"6\",\"func\": summarize_policy_content },\r\n\u00a0 \u00a0 {\"id\": \"7\",\"func\": find_careproviders },\r\n\u00a0 \u00a0 {\"id\": \"8\",\"func\": analyze_sentiment },\r\n\u00a0 \u00a0 {\"id\": \"9\",\"func\": ask_a_question},\r\n]<\/code><\/pre>\n<p><strong>3. The framework file that manages the function API file.<\/strong><\/p>\n<p>We use chromdb for in memory vector DB for the sake of convenience. For better performance, using Azure Cognitive Search service is recommended.<\/p>\n<pre class=\"prettyprint language-py\"><code class=\"language-py\">from pydantic import BaseModel, Field\r\nfrom typing_extensions import Annotated\r\nimport autogen\r\nimport os\r\nfrom dotenv import load_dotenv \u00a0\r\nimport _FunctionFactory_5 as functions\r\n# load llm config\r\nload_dotenv() \u00a0\r\nconfig_list = [{\r\n\u00a0 \u00a0 'model': os.getenv(\"AZURE_OPENAI_MODEL\"),\r\n\u00a0 \u00a0 'api_key': os.getenv(\"AZURE_OPENAI_API_KEY\"),\r\n\u00a0 \u00a0 'base_url': os.getenv(\"AZURE_OPENAI_ENDPOINT\"),\r\n\u00a0 \u00a0 'api_type': 'azure',\r\n\u00a0 \u00a0 'api_version': os.getenv(\"AZURE_OPENAI_API_VERSION\"),\r\n\u00a0 \u00a0 'tags': [\"tool\", \"gpt-4\"]\r\n\u00a0 \u00a0 }]\r\nllm_config={\r\n\u00a0 \u00a0 \"config_list\": config_list,\r\n\u00a0 \u00a0 \"timeout\": 120,\r\n\u00a0 \u00a0 }\r\n# functions dict for lookup used by get_function(function_name)\r\nfunctions_dict = {item[\"func\"].__name__: item[\"func\"] for item in functions.functions_table}\r\n# in memory vector database for function lookup\r\nimport chromadb\r\ndocuments = []\r\nmetadatas = []\r\nids = []\r\n# populate the documents, metadatas and ids for the functions\r\nfor item in functions.functions_table:\r\n\u00a0 \u00a0 documents.append(item[\"func\"].__desc__)\r\n\u00a0 \u00a0 metadatas.append({\"name\": item[\"func\"].__name__})\r\n\u00a0 \u00a0 ids.append(item[\"id\"])\r\n# create the collection and add the documents\r\nclient=chromadb.Client()\r\ncollection = client.create_collection(\"functions\")\r\ncollection.add(\r\n\u00a0 \u00a0 \u00a0 \u00a0 documents=documents, # we embed for you, or bring your own\r\n\u00a0 \u00a0 \u00a0 \u00a0 metadatas=metadatas, # filter on arbitrary metadata!\r\n\u00a0 \u00a0 \u00a0 \u00a0 ids=ids, # must be unique for each doc\r\n)\r\n# function factory to get a function based on the description. the fuction will be called by the user proxy agent\r\nfrom typing import Callable, Any, Dict \u00a0\r\ndef get_function(description: str) -&gt; Callable[..., Any]:\r\n\u00a0 \u00a0 \"\"\"\r\n\u00a0 \u00a0 use the description to find the function based on vector search for now, use the hard coded function map for testing\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 args:\r\n\u00a0 \u00a0 \u00a0 \u00a0 description (str): the description of the function.\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 returns:\r\n\u00a0 \u00a0 \u00a0 \u00a0 Callable[..., Any]: the function.\r\n\u00a0 \u00a0 \"\"\"\r\n\u00a0 \u00a0 results = collection.query(\r\n\u00a0 \u00a0 \u00a0 \u00a0 query_texts=[description],\r\n\u00a0 \u00a0 \u00a0 \u00a0 n_results=1,\r\n\u00a0 \u00a0 \u00a0 \u00a0 # where={\"metadata_field\": \"is_equal_to_this\"}, # optional filter\r\n\u00a0 \u00a0 \u00a0 \u00a0 # where_document={\"$contains\":\"search_string\"} \u00a0# optional filter\r\n\u00a0 \u00a0 )\r\n\u00a0 \u00a0 name = results[\"metadatas\"][0][0][\"name\"]\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 func = functions_dict.get(name, None)\r\n\u00a0 \u00a0 if func is not None:\r\n\u00a0 \u00a0 \u00a0 \u00a0 return func\r\n\u00a0 \u00a0 else:\r\n\u00a0 \u00a0 \u00a0 \u00a0 print(f\"get_function() error: function found for: {description}\")\r\n\u00a0 \u00a0 \u00a0 \u00a0 raise Exception(f\"get_function fail to find function for: {description})\")\r\n# function to register other functions for agent to call, given the description \u00a0 \u00a0\r\n@functions.desc(\"register the function for agent based on the given description\")\r\ndef register_functions(function_description: Annotated[str, \"description of the function to register.\"]) \u00a0-&gt; Annotated[str, \"registration result\"]:\r\n\u00a0 \u00a0 \"\"\"\r\n\u00a0 \u00a0 register the functions based on the description\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 args:\r\n\u00a0 \u00a0 \u00a0 \u00a0 function_description (str): the description of the function to register.\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 returns:\r\n\u00a0 \u00a0 \u00a0 \u00a0 str: registration result\r\n\u00a0 \u00a0 \"\"\"\r\n\u00a0 \u00a0 func = get_function(function_description)\r\n\u00a0 \u00a0 assistant.register_for_llm(name=func.__name__, description=func.__desc__)(func)\r\n\u00a0 \u00a0 user_proxy.register_for_execution(name=func.__name__)(func)\r\n\u00a0 \u00a0 return f\"registering: {func.__name__} for: '{function_description}'\"\r\n# create user agent and assistant agent\r\n\u00a0 \u00a0\r\nuser_system_message = \"\"\"\r\n\u00a0 \u00a0 You are a helpful AI agent. when you talk with assistant, help them to make their response as accurate as possible to the user's requirement. make sure to execute the task in the correct order of the assistant's response.\r\n\u00a0\"\"\"\r\nassistant_system_message = \"\"\"\r\n\u00a0 \u00a0 For coding tasks, only use the functions you have been provided with.\r\n\u00a0 \u00a0 do not generate answer on your own. do not guess.\r\n\u00a0 \u00a0 for tasks that needs to access user local resources, do not generate python code. use the functions provided to you.\r\n\u00a0 \u00a0 if you don't have enough information to execute the task, call the given 'register_functions' function with a brief description e.g. 'get insurance policy'.\r\n\u00a0 \u00a0 if you need to save content to or read content from a file, call register_functions function to register functions that can save to or read from file.\r\n\u00a0 \u00a0 Reply TERMINATE when the task is done.\r\n\"\"\"\r\nimport typing;\r\nassistant = None\r\nuser_proxy = None\r\ndef Create_Agents( ) -&gt; typing.Tuple[autogen.UserProxyAgent, autogen.AssistantAgent]:\r\n\u00a0 \u00a0 global assistant\r\n\u00a0 \u00a0 global user_proxy\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 assistant = autogen.AssistantAgent(\r\n\u00a0 \u00a0 \u00a0 \u00a0 name=\"assistant\",\r\n\u00a0 \u00a0 \u00a0 \u00a0 system_message=assistant_system_message,\r\n\u00a0 \u00a0 \u00a0 \u00a0 llm_config=llm_config,\r\n\u00a0 \u00a0 )\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 user_proxy = autogen.UserProxyAgent(\r\n\u00a0 \u00a0 \u00a0 \u00a0 name=\"User\",\r\n\u00a0 \u00a0 \u00a0 \u00a0 llm_config=False,\r\n\u00a0 \u00a0 \u00a0 \u00a0 is_termination_msg=lambda msg: msg.get(\"content\") is not None and \"TERMINATE\" in msg[\"content\"],\r\n\u00a0 \u00a0 \u00a0 \u00a0 system_message=user_system_message,\r\n\u00a0 \u00a0 \u00a0 \u00a0 human_input_mode=\"NEVER\",\r\n\u00a0 \u00a0 \u00a0 \u00a0 max_consecutive_auto_reply=12,\r\n\u00a0 \u00a0 \u00a0 \u00a0 code_execution_config={\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \"work_dir\": \"coding\",\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \"use_docker\": False,\r\n\u00a0 \u00a0 \u00a0 \u00a0 }, \u00a0# Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.\r\n\u00a0 \u00a0 )\r\n\u00a0 \u00a0 # register the fundamental functions\r\n\u00a0 \u00a0 assistant.register_for_llm(name=\"register_functions\", description=register_functions.__desc__)(register_functions)\r\n\u00a0 \u00a0 user_proxy.register_for_execution(name=\"register_functions\")(register_functions)\r\n\u00a0 \u00a0 return user_proxy, assistant\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0\r\n# reset the agents to their initial state\r\ndef Reset_Agents():\r\n\u00a0 \u00a0 global user_proxy\r\n\u00a0 \u00a0 global assistant\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 user_proxy.clear_history()\r\n\u00a0 \u00a0 user_proxy.function_map.clear()\r\n\u00a0 \u00a0 assistant.llm_config=llm_config\r\n\u00a0 \u00a0 # register the fundamental functions\r\n\u00a0 \u00a0 assistant.register_for_llm(name=\"register_functions\", description=register_functions.__desc__)(register_functions)\r\n\u00a0 \u00a0 user_proxy.register_for_execution(name=\"register_functions\")(register_functions)\r\n# test the function factory\r\nif __name__ == \"__main__\":\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 print(\"functions table:\")\r\n\u00a0 \u00a0 for item in functions.functions_table:\r\n\u00a0 \u00a0 \u00a0 \u00a0 print(f\"function: {item['func'].__name__}, id: {item['id']}\")\r\n\u00a0 \u00a0 \u00a0 \u00a0\r\n\u00a0 \u00a0 print()\r\n\u00a0 \u00a0 print(\"functions_dict:\")\r\n\u00a0 \u00a0 for key, value in functions_dict.items():\r\n\u00a0 \u00a0 \u00a0 \u00a0 print(f\"key: {key}, value: {value}\")\r\n\u00a0 \u00a0 \u00a0 \u00a0\r\n\u00a0 \u00a0 print()\r\n\u00a0 \u00a0 print(\"test get function find_careproviders\")\r\n\u00a0 \u00a0 func = functions_dict.get(\"find_careproviders\")\r\n\u00a0 \u00a0 print(func.__name__) \u00a0\r\n\u00a0 \u00a0 print()\r\n\u00a0 \u00a0 print(\"test register_function\")\r\n\u00a0 \u00a0 user_proxy, assistant = Create_Agents() # to initialize the agents\r\n\u00a0 \u00a0 for item in functions.functions_table:\r\n\u00a0 \u00a0 \u00a0 \u00a0 register_functions(item[\"func\"].__desc__)\r\n\u00a0 \u00a0 \u00a0 \u00a0 print(f\"function: {item['func'].__name__}, id: {item['id']}\")\r\n\u00a0 \u00a0 print()\r\n\u00a0 \u00a0 print(\"test user_proxy.function_map\")\r\n\u00a0 \u00a0 for item in user_proxy.function_map:\r\n\u00a0 \u00a0 \u00a0 \u00a0 print(f\"key: {item}\")\r\n\u00a0 \u00a0 \u00a0 \u00a0\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 while True:\r\n\u00a0 \u00a0 \u00a0 \u00a0 user_input = input(\"Enter your input: \")\r\n\u00a0 \u00a0 \u00a0 \u00a0 if user_input == \"exit\":\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 break\r\n\u00a0 \u00a0 \u00a0 \u00a0 chat_result = user_proxy.initiate_chat(\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 assistant,\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 message=user_input, \u00a0\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 max_turns=10,\r\n\u00a0 \u00a0 \u00a0 \u00a0 )\r\n\u00a0 \u00a0 \u00a0 \u00a0 print(\"chat complete\")\r\n\u00a0 \u00a0 \u00a0 \u00a0\r\n\u00a0 \u00a0 \u00a0 \u00a0 Reset_Agents()<\/code><\/pre>\n<p><strong>The main file:\n<\/strong><\/p>\n<pre class=\"prettyprint language-py\"><code class=\"language-py\">import autogen\r\nimport _autogenRAG_5 as autogenRAG\r\nuser_proxy, assistant = autogenRAG.Create_Agents()\r\n# take user input prompt and call the assistant\r\nwhile True:\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 user_input = input(\"Enter your input: \")\r\n\u00a0 \u00a0 if user_input == \"exit\":\r\n\u00a0 \u00a0 \u00a0 \u00a0 break\r\n\u00a0 \u00a0 chat_result = user_proxy.initiate_chat(\r\n\u00a0 \u00a0 \u00a0 \u00a0 assistant,\r\n\u00a0 \u00a0 \u00a0 \u00a0 message=user_input, \u00a0\r\n\u00a0 \u00a0 \u00a0 \u00a0 max_turns=12,\r\n\u00a0 \u00a0 )\r\n\u00a0 \u00a0 print(\"chat complete\")\r\n\u00a0 \u00a0\r\n\u00a0 \u00a0 autogenRAG.Reset_Agents()<\/code><\/pre>\n<h3>At runtime, you can provide this prompt as an example:<\/h3>\n<p><strong>Read file prompt-find-benefits.txt. Take the content as user input and execute it. When finish the task, reply TERMINATE<\/strong>.<\/p>\n<p>The content of the prompt-find-benefits.txt is this:<\/p>\n<p><em><strong>I have a health insurance account with me\u00a0as the primary policy holder (name linkai yu). I would like to find out the benefits of my policy.<\/strong><\/em><\/p>\n<p><em><strong>Summarize it in one paragraph, and then save the summary to file c:\\\\temp\\\\output_benefit_summary.txt.<\/strong><\/em><\/p>\n<p>You should see the screen output like this:<\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42305 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-white-and-green-text-descr-1-1024x403.png\" alt=\"A computer screen with white and green text Description automatically generated\" width=\"1024\" height=\"403\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-white-and-green-text-descr-1-1024x403.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-white-and-green-text-descr-1-300x118.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-white-and-green-text-descr-1-768x302.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-white-and-green-text-descr-1.png 1069w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42307 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-4-1024x523.png\" alt=\"A computer screen with text Description automatically generated\" width=\"1024\" height=\"523\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-4-1024x523.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-4-300x153.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-4-768x392.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-4.png 1047w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42309 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-screen-description-aut-1-1024x697.png\" alt=\"A screenshot of a computer screen Description automatically generated\" width=\"1024\" height=\"697\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-screen-description-aut-1-1024x697.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-screen-description-aut-1-300x204.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-screen-description-aut-1-768x523.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-screen-description-aut-1.png 1063w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42311 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-2-1024x530.png\" alt=\"A computer screen with text on it Description automatically generated\" width=\"1024\" height=\"530\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-2-1024x530.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-2-300x155.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-2-768x398.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-2.png 1051w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42312 size-full\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-a-computer-program-desc-1.png\" alt=\"A computer screen shot of a computer program Description automatically generated\" width=\"955\" height=\"574\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-a-computer-program-desc-1.png 955w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-a-computer-program-desc-1-300x180.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-a-computer-program-desc-1-768x462.png 768w\" sizes=\"(max-width: 955px) 100vw, 955px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42313 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-3-1024x526.png\" alt=\"A computer screen with text on it Description automatically generated\" width=\"1024\" height=\"526\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-3-1024x526.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-3-300x154.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-3-768x394.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-on-it-description-aut-3.png 1060w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42314 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-5-1024x555.png\" alt=\"A computer screen with text Description automatically generated\" width=\"1024\" height=\"555\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-5-1024x555.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-5-300x162.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-5-768x416.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-5.png 1047w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42315 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-6-1024x524.png\" alt=\"A computer screen with text Description automatically generated\" width=\"1024\" height=\"524\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-6-1024x524.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-6-300x153.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-6-768x393.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-6.png 1056w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42316 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-2-1024x556.png\" alt=\"A computer screen shot of text Description automatically generated\" width=\"1024\" height=\"556\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-2-1024x556.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-2-300x163.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-2-768x417.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-2.png 1057w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42317 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-7-1024x524.png\" alt=\"A computer screen with text Description automatically generated\" width=\"1024\" height=\"524\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-7-1024x524.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-7-300x154.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-7-768x393.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-with-text-description-automatic-7.png 1057w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42318 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-3-1024x550.png\" alt=\"A computer screen shot of text Description automatically generated\" width=\"1024\" height=\"550\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-3-1024x550.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-3-300x161.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-3-768x412.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-computer-screen-shot-of-text-description-automa-3.png 1056w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><img decoding=\"async\" class=\"alignnone wp-image-42319 size-large\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-program-description-au-2-1024x575.png\" alt=\"A screenshot of a computer program Description automatically generated\" width=\"1024\" height=\"575\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-program-description-au-2-1024x575.png 1024w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-program-description-au-2-300x168.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-program-description-au-2-768x431.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/a-screenshot-of-a-computer-program-description-au-2.png 1074w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat1.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-42329\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat1.png\" alt=\"Image chat1\" width=\"936\" height=\"717\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat1.png 936w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat1-300x230.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat1-768x588.png 768w\" sizes=\"(max-width: 936px) 100vw, 936px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat2.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-42328\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat2.png\" alt=\"Image chat2\" width=\"936\" height=\"449\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat2.png 936w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat2-300x144.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat2-768x368.png 768w\" sizes=\"(max-width: 936px) 100vw, 936px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat3.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-42327\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat3.png\" alt=\"Image chat3\" width=\"936\" height=\"801\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat3.png 936w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat3-300x257.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat3-768x657.png 768w\" sizes=\"(max-width: 936px) 100vw, 936px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat4.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-42326\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat4.png\" alt=\"Image chat4\" width=\"925\" height=\"453\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat4.png 925w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat4-300x147.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2024\/09\/chat4-768x376.png 768w\" sizes=\"(max-width: 925px) 100vw, 925px\" \/><\/a><\/p>\n<p>The whole code solution is available from:<\/p>\n<p><a href=\"https:\/\/github.com\/linkaiyu\/AutogenRAG\">linkaiyu\/AutogenRAG: autogen with RAG capabilities by dynamically invoking custom functions for data IO (github.com)<\/a><\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Linkai Yu explores the retrieval-augemented generation pattern with Azure OpenAI Assistants API. The Retrieval-Augmented Generation (RAG) pattern is the standard for integrating ground AI with local data. One powerful way to implement RAG is through the function call or tool call feature in the Function Call API. When examining the samples provided by the Azure [&hellip;]<\/p>\n","protected":false},"author":582,"featured_media":38844,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[14],"tags":[10650,10659],"class_list":["post-42295","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-azure-openai","tag-rag"],"acf":[],"blog_post_summary":"<p>Linkai Yu explores the retrieval-augemented generation pattern with Azure OpenAI Assistants API. The Retrieval-Augmented Generation (RAG) pattern is the standard for integrating ground AI with local data. One powerful way to implement RAG is through the function call or tool call feature in the Function Call API. When examining the samples provided by the Azure [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/42295","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/users\/582"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/comments?post=42295"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/42295\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media\/38844"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media?parent=42295"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/categories?post=42295"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/tags?post=42295"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}