Architecture<\/span>\u00a0<\/span><\/h3>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/02\/Screenshot-2025-02-25-153758-1024x431.png\")
<\/a><\/p>\nThe\u00a0LLMAgentOps\u00a0architecture might be constructed using the following key components divided into two phases like DevOps \/ MLOps \/ LLMOps development and deployment phases:<\/p>\n
\n- LLM Agent Development Phase (inner loop)<\/strong>:\n
\n- Agent Architecture: Designing the agent architecture for the LLM Agent based solution. For this sample we have used\u00a0Semantic Kernel\u00a0development kit by using\u00a0Python\u00a0programming language.<\/li>\n
- Experimentation & Evaluation: Experimentation and Evaluation of the LLM Agent based solution. Where the experimentation is done using\u00a0console\u00a0or\u00a0UI\u00a0or in\u00a0batch\u00a0mode and evaluation is done using\u00a0LLM as Judge\u00a0and\u00a0Human Evaluation.<\/li>\n<\/ul>\n<\/li>\n
- LLM Agent Deployment Phase (outer loop)<\/strong>:\n
\n- GitHub Actions: Continuous Integration, Evaluation and Deployment of the LLM Agent based solution with addition of\u00a0Continuous Security<\/strong>\u00a0for security checks of the LLM Agents.<\/li>\n
- Deployment: Deployment of the LLM Agent based solution in\u00a0local\u00a0or\u00a0cloud\u00a0environment.<\/li>\n
- Monitoring: Monitoring the LLM Agent based solution for data collection, performance and other metrics.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Source Code Structure<\/h1>\n
The\u00a0source code\u00a0of LLMAgentOps application might be structured in such a way that it can be easily developed and maintained by data scientists and developers together with following key concepts of dividing the code into two parts –\u00a0core\u00a0and\u00a0ops:<\/p>\n
\n- Core<\/strong>: The LLM Agent core implementation code.\n
\n- Agents Base Class<\/strong>: The\u00a0base class\u00a0for the agents.<\/li>\n
- Agents<\/strong>: All the\u00a0agents\u00a0with their specific prompts and descriptions. Example:\u00a0Observe Agent.<\/li>\n
- Code Execute Agent (optional)<\/strong>: The\u00a0code execute agent\u00a0is an agent that can join the group of agents, but it will execute the code and return the result, instead of using LLM for generating response like other agents.<\/li>\n
- Group Chat Selection Logic<\/strong>: The\u00a0group chat selection logic\u00a0is used to select the appropriate next agent based on the current state of the conversation.<\/li>\n
- Group Chat Termination Logic<\/strong>: The\u00a0group chat termination logic\u00a0is used to terminate the conversation based on the current state of the conversation or maximum number of turns.<\/li>\n
- Group Chat<\/strong>: The\u00a0group chat\u00a0contains the group chat client that can serve the conversation between the user and the agents.<\/li>\n<\/ul>\n<\/li>\n
- Ops<\/strong>: The operational code for the LLM Agent based solution.\n
\n- Observability<\/strong>: The\u00a0observability code\u00a0contains the code for logging and monitoring the agents. OpenTelemetry\u00a0can be used for logging and monitoring.<\/li>\n
- Application Specific Codes (optional)<\/strong>: There could application specific code, like code for interacting with database or code for integrating with other systems.<\/li>\n
- Deployment<\/strong>: The deployment code contains the code for deploying the agents in local or cloud environment. In this sample the code is provided for deploying the agents in Azure Web App Service. The deployment code will be:\n
\n- Source Module: core implementation of the agents and group chat.<\/li>\n
- REST API Based App: REST API based app for calling the agents and getting the response.<\/li>\n
- Dockefile: for building the image of the entire application.<\/li>\n
- Requirements\u00a0file for the dependencies.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n
- Experimentation: <\/strong>The code related to performing experimentation in Console\u00a0or\u00a0User Interface\u00a0or in\u00a0Batch\u00a0mode.<\/li>\n
- Evaluation: <\/strong>The evaluation related code for\u00a0LLM as Judge\u00a0and\u00a0Human Evaluation.<\/li>\n
- Security:<\/strong> The code for the security checks of the LLM Agent based solution.<\/li>\n<\/ul>\n
Experimentation<\/h1>\n
The\u00a0experimentation\u00a0setup by using\u00a0could be more complex and granular. The experimentation process may involve starting from defining the problem => data collection => LLM agent design => experiments. In this toolkit we have demonstrated how experiments can be done using three following modes:<\/p>\n
\n- Experimentation with Console<\/strong><\/li>\n<\/ol>\n
The\u00a0Console\u00a0based experimentation involves running the LLM Agent in the console and interacting with it using a text-based interface.<\/p>\n
Note<\/strong>: The console-based experimentation is for initial\u00a0exploration\u00a0stage and does not store any data for evaluation.<\/p>\nSample experiment:\n![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/03\/Screenshot-2025-02-25-164956.png\")
<\/a><\/p>\n2.Experimentation with User Interface (UI)<\/strong><\/p>\nThe user interface drive experimentation can be achieved using\u00a0Chainlit<\/a> where the LLM Agents can be interacted using a conversational interface. The Chainlit provides a more interactive and user-friendly experience for the LLM Agents based solution. It not only stores the conversation data for evaluation but also provides a way to provide\u00a0Human Feedback\u00a0for the overall conversation and individual agents.<\/p>\nNote<\/strong>: The UI based experimentation is\u00a0experiment\u00a0stage and it can be provided to\u00a0Human Evaluator\u00a0for feedback. The collected data can be viewed by opening the\u00a0SQLite file in an SQLite browser.<\/p>\nSample experiment:<\/p>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/03\/Screenshot-2025-02-25-170449.jpg\")
<\/a><\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/03\/Screenshot-2025-02-25-170553.jpg\")
<\/a><\/p>\n3. Experimentation in Batch<\/strong><\/p>\nBatch-based experimentation involves running the LLM Agent in batch mode to process many queries and evaluate the solution’s performance. The batch experimentation can be used to evaluate the accuracy, efficiency, and scalability of the LLM Agents based solution.<\/p>\n
Note<\/strong>: The batch-based experimentation is for\u00a0continuous evaluation\u00a0stage, and it can be used to evaluate the performance of the LLM Agents based solution.<\/p>\nEvaluation<\/h1>\n
Evaluation is the process of evaluating the performance of the LLM Agents based solution, that will help in the decision-making process of the LLM Agents based solution. In this toolkit we have demonstrated how evaluation can be done in two following modes:<\/p>\n
\n- Human Evaluation<\/strong><\/li>\n<\/ol>\n
The\u00a0Human Evaluation\u00a0is the process of evaluating the performance of the LLM Agents based solution by providing the conversational interface to the\u00a0Human Evaluator. The\u00a0Human Evaluator\u00a0will interact with the LLM Agents using a conversational interface and provide feedback on the overall conversation and individual agents.<\/p>\n
This can be achieved by running the Experiment in the UI mode and providing the\u00a0Chainlit\u00a0based interface to the\u00a0Human Evaluator.<\/p>\n
Sample Evaluation:\n![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/02\/Picture7.png\")
<\/a><\/p>\n2. LLM as Judge<\/strong><\/p>\nThe LLM as Judge is the process of evaluating the performance of the LLM Agents using another LLM Agent as a judge. For this evaluations Azure AI Foundry Service can be used. For more details, refer to the documentation<\/a>.<\/p>\nThis can be achieved by running the experiment in the batch mode.<\/p>\n
Sample Azure AI Foundry Evaluation Result:<\/p>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/03\/Screenshot-2025-02-25-170318.jpg\")
<\/a><\/p>\nSecurity Scanning<\/h1>\n
Security Scanning is the process of ensuring the security of the LLM Agents based solution. Agents are going to write \/ execute code, browse the web, and interact with databases, hence security is a key concern and must be designed and implemented from the beginning.<\/p>\n
The security scan of LLM Agents can be performed using following tools:<\/p>\n
\n- Risk Identification Tool for generative AI (PyRIT)<\/a><\/li>\n
- LLM Guard – The Security Toolkit for LLM Interactions<\/a><\/li>\n<\/ul>\n
In this toolkit we have demonstrated how to enable Continuous Security Scan<\/strong> using LLM Guard.<\/p>\nSample Security Scanning Result:<\/p>\n
==================Summary=====================<\/em><\/p>\nAgent Error avg score for scan BanTopics: 0.6<\/em><\/p>\nAgent Observe avg score for scan BanTopics: 1.0<\/em><\/p>\nAgent Verify avg score for scan BanTopics: 0.8<\/em><\/p>\nAgent Select avg score for scan BanTopics: 1.0<\/em><\/p>\nOverall avg score for scan BanTopics: 0.85<\/em><\/p>\n===============================================<\/em><\/p>\nEngineering Fundamentals<\/h1>\n
<\/p>\n
The following engineering fundamentals must be considered while designing and developing the LLM Agent based solution:<\/p>\n
\n- GitHub Actions<\/a>: \u00a0for continuous integration, continuous evaluation, continuous deployment and continuous security scanning of the LLM Agent based solution.<\/li>\n
- Dev Containers<\/a>: to enable full-featured development environment for LLM Agents, where all required tools are installed and configured to perform rapid experimentation, evaluation and testing.<\/li>\n
- Unit Testing of Agents<\/a>: for testing the Python codes locally with unit test cases.<\/li>\n<\/ol>\n
Toolkit Repository<\/h1>\n
The toolkit repository Azure-Samples\/llm-agent-ops-toolkit-sk<\/a> contains a sample use case of\u00a0MySQL Copilot<\/strong>, where user can interact with the solution to retrieve data from a MySQL database by providing a natural language query. The solution uses agentic approach, where LLM Agents will process the user query, generate SQL queries, execute the queries on the MySQL database, and return the results to the user.<\/p>\nThis has been implemented using the concept of\u00a0StateFlow<\/strong>\u00a0(a Finite State Machine FSM based LLM workflow) using\u00a0Semantic Kernel<\/a>\u00a0agents. This implementation is equivalent to\u00a0AutoGen Selector Group Chat Pattern with custom selector function<\/a>.<\/p>\nFor more details on\u00a0StateFlow\u00a0refer the research paper –\u00a0StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows<\/a>.<\/p>\nExtending the Toolkit<\/h1>\n
This toolkit can be used by forking this repository and replacing the\u00a0MySQL Copilot\u00a0with any other LLM Agent based solution or it can be further enhanced for a specific use case.<\/p>\n
Conclusion<\/h2>\n
<\/a><\/p>\nThe\u00a0LLMAgentOps\u00a0architecture might be constructed using the following key components divided into two phases like DevOps \/ MLOps \/ LLMOps development and deployment phases:<\/p>\n
- \n
- LLM Agent Development Phase (inner loop)<\/strong>:\n
- \n
- Agent Architecture: Designing the agent architecture for the LLM Agent based solution. For this sample we have used\u00a0Semantic Kernel\u00a0development kit by using\u00a0Python\u00a0programming language.<\/li>\n
- Experimentation & Evaluation: Experimentation and Evaluation of the LLM Agent based solution. Where the experimentation is done using\u00a0console\u00a0or\u00a0UI\u00a0or in\u00a0batch\u00a0mode and evaluation is done using\u00a0LLM as Judge\u00a0and\u00a0Human Evaluation.<\/li>\n<\/ul>\n<\/li>\n
- LLM Agent Deployment Phase (outer loop)<\/strong>:\n
- \n
- GitHub Actions: Continuous Integration, Evaluation and Deployment of the LLM Agent based solution with addition of\u00a0Continuous Security<\/strong>\u00a0for security checks of the LLM Agents.<\/li>\n
- Deployment: Deployment of the LLM Agent based solution in\u00a0local\u00a0or\u00a0cloud\u00a0environment.<\/li>\n
- Monitoring: Monitoring the LLM Agent based solution for data collection, performance and other metrics.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Source Code Structure<\/h1>\n
The\u00a0source code\u00a0of LLMAgentOps application might be structured in such a way that it can be easily developed and maintained by data scientists and developers together with following key concepts of dividing the code into two parts –\u00a0core\u00a0and\u00a0ops:<\/p>\n
- \n
- Core<\/strong>: The LLM Agent core implementation code.\n
- \n
- Agents Base Class<\/strong>: The\u00a0base class\u00a0for the agents.<\/li>\n
- Agents<\/strong>: All the\u00a0agents\u00a0with their specific prompts and descriptions. Example:\u00a0Observe Agent.<\/li>\n
- Code Execute Agent (optional)<\/strong>: The\u00a0code execute agent\u00a0is an agent that can join the group of agents, but it will execute the code and return the result, instead of using LLM for generating response like other agents.<\/li>\n
- Group Chat Selection Logic<\/strong>: The\u00a0group chat selection logic\u00a0is used to select the appropriate next agent based on the current state of the conversation.<\/li>\n
- Group Chat Termination Logic<\/strong>: The\u00a0group chat termination logic\u00a0is used to terminate the conversation based on the current state of the conversation or maximum number of turns.<\/li>\n
- Group Chat<\/strong>: The\u00a0group chat\u00a0contains the group chat client that can serve the conversation between the user and the agents.<\/li>\n<\/ul>\n<\/li>\n
- Ops<\/strong>: The operational code for the LLM Agent based solution.\n
- \n
- Observability<\/strong>: The\u00a0observability code\u00a0contains the code for logging and monitoring the agents. OpenTelemetry\u00a0can be used for logging and monitoring.<\/li>\n
- Application Specific Codes (optional)<\/strong>: There could application specific code, like code for interacting with database or code for integrating with other systems.<\/li>\n
- Deployment<\/strong>: The deployment code contains the code for deploying the agents in local or cloud environment. In this sample the code is provided for deploying the agents in Azure Web App Service. The deployment code will be:\n
- \n
- Source Module: core implementation of the agents and group chat.<\/li>\n
- REST API Based App: REST API based app for calling the agents and getting the response.<\/li>\n
- Dockefile: for building the image of the entire application.<\/li>\n
- Requirements\u00a0file for the dependencies.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n
- Experimentation: <\/strong>The code related to performing experimentation in Console\u00a0or\u00a0User Interface\u00a0or in\u00a0Batch\u00a0mode.<\/li>\n
- Evaluation: <\/strong>The evaluation related code for\u00a0LLM as Judge\u00a0and\u00a0Human Evaluation.<\/li>\n
- Security:<\/strong> The code for the security checks of the LLM Agent based solution.<\/li>\n<\/ul>\n
Experimentation<\/h1>\n
The\u00a0experimentation\u00a0setup by using\u00a0could be more complex and granular. The experimentation process may involve starting from defining the problem => data collection => LLM agent design => experiments. In this toolkit we have demonstrated how experiments can be done using three following modes:<\/p>\n
- \n
- Experimentation with Console<\/strong><\/li>\n<\/ol>\n
The\u00a0Console\u00a0based experimentation involves running the LLM Agent in the console and interacting with it using a text-based interface.<\/p>\n
Note<\/strong>: The console-based experimentation is for initial\u00a0exploration\u00a0stage and does not store any data for evaluation.<\/p>\n
Sample experiment:\n
<\/a><\/p>\n2.Experimentation with User Interface (UI)<\/strong><\/p>\n
The user interface drive experimentation can be achieved using\u00a0Chainlit<\/a> where the LLM Agents can be interacted using a conversational interface. The Chainlit provides a more interactive and user-friendly experience for the LLM Agents based solution. It not only stores the conversation data for evaluation but also provides a way to provide\u00a0Human Feedback\u00a0for the overall conversation and individual agents.<\/p>\n
Note<\/strong>: The UI based experimentation is\u00a0experiment\u00a0stage and it can be provided to\u00a0Human Evaluator\u00a0for feedback. The collected data can be viewed by opening the\u00a0SQLite file in an SQLite browser.<\/p>\n
Sample experiment:<\/p>\n
<\/a><\/p>\n<\/a><\/p>\n3. Experimentation in Batch<\/strong><\/p>\n
Batch-based experimentation involves running the LLM Agent in batch mode to process many queries and evaluate the solution’s performance. The batch experimentation can be used to evaluate the accuracy, efficiency, and scalability of the LLM Agents based solution.<\/p>\n
Note<\/strong>: The batch-based experimentation is for\u00a0continuous evaluation\u00a0stage, and it can be used to evaluate the performance of the LLM Agents based solution.<\/p>\n
Evaluation<\/h1>\n
Evaluation is the process of evaluating the performance of the LLM Agents based solution, that will help in the decision-making process of the LLM Agents based solution. In this toolkit we have demonstrated how evaluation can be done in two following modes:<\/p>\n
- \n
- Human Evaluation<\/strong><\/li>\n<\/ol>\n
The\u00a0Human Evaluation\u00a0is the process of evaluating the performance of the LLM Agents based solution by providing the conversational interface to the\u00a0Human Evaluator. The\u00a0Human Evaluator\u00a0will interact with the LLM Agents using a conversational interface and provide feedback on the overall conversation and individual agents.<\/p>\n
This can be achieved by running the Experiment in the UI mode and providing the\u00a0Chainlit\u00a0based interface to the\u00a0Human Evaluator.<\/p>\n
Sample Evaluation:\n
<\/a><\/p>\n2. LLM as Judge<\/strong><\/p>\n
The LLM as Judge is the process of evaluating the performance of the LLM Agents using another LLM Agent as a judge. For this evaluations Azure AI Foundry Service can be used. For more details, refer to the documentation<\/a>.<\/p>\n
This can be achieved by running the experiment in the batch mode.<\/p>\n
Sample Azure AI Foundry Evaluation Result:<\/p>\n
<\/a><\/p>\nSecurity Scanning<\/h1>\n
Security Scanning is the process of ensuring the security of the LLM Agents based solution. Agents are going to write \/ execute code, browse the web, and interact with databases, hence security is a key concern and must be designed and implemented from the beginning.<\/p>\n
The security scan of LLM Agents can be performed using following tools:<\/p>\n
- \n
- Risk Identification Tool for generative AI (PyRIT)<\/a><\/li>\n
- LLM Guard – The Security Toolkit for LLM Interactions<\/a><\/li>\n<\/ul>\n
In this toolkit we have demonstrated how to enable Continuous Security Scan<\/strong> using LLM Guard.<\/p>\n
Sample Security Scanning Result:<\/p>\n
==================Summary=====================<\/em><\/p>\n
Agent Error avg score for scan BanTopics: 0.6<\/em><\/p>\n
Agent Observe avg score for scan BanTopics: 1.0<\/em><\/p>\n
Agent Verify avg score for scan BanTopics: 0.8<\/em><\/p>\n
Agent Select avg score for scan BanTopics: 1.0<\/em><\/p>\n
Overall avg score for scan BanTopics: 0.85<\/em><\/p>\n
===============================================<\/em><\/p>\n
Engineering Fundamentals<\/h1>\n
<\/p>\n
The following engineering fundamentals must be considered while designing and developing the LLM Agent based solution:<\/p>\n
- \n
- GitHub Actions<\/a>: \u00a0for continuous integration, continuous evaluation, continuous deployment and continuous security scanning of the LLM Agent based solution.<\/li>\n
- Dev Containers<\/a>: to enable full-featured development environment for LLM Agents, where all required tools are installed and configured to perform rapid experimentation, evaluation and testing.<\/li>\n
- Unit Testing of Agents<\/a>: for testing the Python codes locally with unit test cases.<\/li>\n<\/ol>\n
Toolkit Repository<\/h1>\n
The toolkit repository Azure-Samples\/llm-agent-ops-toolkit-sk<\/a> contains a sample use case of\u00a0MySQL Copilot<\/strong>, where user can interact with the solution to retrieve data from a MySQL database by providing a natural language query. The solution uses agentic approach, where LLM Agents will process the user query, generate SQL queries, execute the queries on the MySQL database, and return the results to the user.<\/p>\n
This has been implemented using the concept of\u00a0StateFlow<\/strong>\u00a0(a Finite State Machine FSM based LLM workflow) using\u00a0Semantic Kernel<\/a>\u00a0agents. This implementation is equivalent to\u00a0AutoGen Selector Group Chat Pattern with custom selector function<\/a>.<\/p>\n
For more details on\u00a0StateFlow\u00a0refer the research paper –\u00a0StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows<\/a>.<\/p>\n
Extending the Toolkit<\/h1>\n
This toolkit can be used by forking this repository and replacing the\u00a0MySQL Copilot\u00a0with any other LLM Agent based solution or it can be further enhanced for a specific use case.<\/p>\n
Conclusion<\/h2>\n
- Dev Containers<\/a>: to enable full-featured development environment for LLM Agents, where all required tools are installed and configured to perform rapid experimentation, evaluation and testing.<\/li>\n
- LLM Guard – The Security Toolkit for LLM Interactions<\/a><\/li>\n<\/ul>\n
- Risk Identification Tool for generative AI (PyRIT)<\/a><\/li>\n
- Human Evaluation<\/strong><\/li>\n<\/ol>\n
- Evaluation: <\/strong>The evaluation related code for\u00a0LLM as Judge\u00a0and\u00a0Human Evaluation.<\/li>\n
- Application Specific Codes (optional)<\/strong>: There could application specific code, like code for interacting with database or code for integrating with other systems.<\/li>\n
- Agents<\/strong>: All the\u00a0agents\u00a0with their specific prompts and descriptions. Example:\u00a0Observe Agent.<\/li>\n
- Agents Base Class<\/strong>: The\u00a0base class\u00a0for the agents.<\/li>\n
- GitHub Actions: Continuous Integration, Evaluation and Deployment of the LLM Agent based solution with addition of\u00a0Continuous Security<\/strong>\u00a0for security checks of the LLM Agents.<\/li>\n