Why build it yourself?<\/strong><\/h2>\nGiven that GitHub Copilot offers a similar preview feature, one might ask why we chose to build our own solution. The key reasons are control and cost; we have full control over Developer (System) Messages, and the option to include additional task-definition data to the system and control every single step of the way + We only pay for token usage, rather than a per-user subscription. Finally, we see the quality of the reviews being much better.<\/p>\n
How it all works (Building blocks and flow)<\/strong><\/h2>\nBelow are the building blocks and the flow of the AI Agent (each of the 6 steps is explained in detail below)<\/p>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/03\/Steps.png\")
<\/a><\/p>\nStep 1: Developer creates Pull Request<\/strong><\/h2>\nWhen the Developer creates their Pull Request and marks it as ready a GitHub Webhook triggers and informs the AI Agent that it is time to do a review.<\/p>\n
Step 2: Get the Pull Request<\/strong><\/h2>\nThe AI Agent now determines, based on its setting what type of Review it should do and gathers the PR Diff Content (what code the developer has added, changed, and removed)<\/p>\n
Here is how we retrieve the PR diff using the GitHub API<\/em><\/p>\npublic async Task<string> GetPrDiff(GitHubClient client, string owner, string repo, int pullRequestNumber)\r\n{\r\n\u00a0 \u00a0 var diffUrl = $\"https:\/\/api.github.com\/repos\/{owner}\/{repo}\/pulls\/{pullRequestNumber}\";\r\n\u00a0 \u00a0 var diff = await client.Connection.Get<string>(\r\n\u00a0 \u00a0 \u00a0 \u00a0 new Uri(diffUrl),\r\n\u00a0 \u00a0 \u00a0 \u00a0 new Dictionary<string, string>(),\r\n\u00a0 \u00a0 \u00a0 \u00a0 \"application\/vnd.github.v3.diff\"\r\n );\r\n\r\n\u00a0 \u00a0 return diff.Body;\r\n}<\/code><\/pre>\nStep 3: Get Task Definition<\/strong><\/h2>\nBeyond the Code, we also want to feed the AI with a title and description of what task, linked to the PR, that the developer set out to achieve, in order for the AI to evaluate if the code actually accomplishes the job.<\/p>\n
Step 4: Conduct the AI Review (+ optional Step 5)<\/strong><\/h2>\nWe now have all the data needed for the review, we create our Semantic Kernel object with o3-mini-high for Azure Open AI Services as backend.<\/p>\n
var kernelBuilder = Kernel.CreateBuilder();\r\n\r\nvar httpClient = new HttpClient\r\n\r\n{\r\n\r\nTimeout = TimeSpan.FromMinutes(5)\r\n\r\n};\r\n\r\nkernelBuilder.AddAzureOpenAIChatCompletion(\"o3-mini\", endpoint, apiKey, httpClient: httpClient);\r\n\r\nvar kernel = kernelBuilder.Build();<\/code><\/pre>\nOur Agent<\/p>\n
var agent = new ChatCompletionAgent\r\n\r\n{\r\n\r\nName = \"RelewisePrReviewer\",\r\n\r\nKernel = kernel,\r\n\r\nInstructions = \"\"\"\r\n\r\nYou are a highly experienced and extremely thorough C# code reviewer.\r\n\r\nYour task is to review the provided GitHub PR diff.\r\n\r\nYou will particularly pay attention to logical and performance bugs and\r\n\r\nregressions, and generally be very meticulous in determining what this PR\r\n\r\nintroduces in terms of new and\/or changed features and\/or behaviors.\r\n\r\nShould you at any point discover that it would be beneficial to you as the\r\n\r\nreviewer to see additional types, you may at any point continue to ask me for\r\n\r\nadditional types, and then I may provide them to you.\r\n\r\n\"\"\",\r\n\r\nArguments = new KernelArguments(new AzureOpenAIPromptExecutionSettings\r\n\r\n{\r\n\r\nReasoningEffort = ChatReasoningEffortLevel.High,\r\n\r\nResponseFormat = typeof(Review)\r\n\r\n})\r\n\r\n};<\/code><\/pre>\nNote that we use o3-Mini’\u00a0ReasoningEffortLevel and the\u00a0ResponseFormat\u00a0is set so instead of a normal response we get\u00a0structured output<\/a>\u00a0that reduce the Developer (System) Message and allow us to get both the review and additional requested files back in a single request.<\/p>\nThe Structured Output class is as follows.<\/em><\/p>\npublic class Review\r\n{\r\n [Description(\"A Short, concise but sufficient summary of what this PR introduces\/changes, and any general observations you find relevant in the context of the review, focus on functional changes that make a difference to the users of the system\")]\r\n\u00a0 \u00a0 public required string Summary { get; init; }\r\n\r\n [Description(\"You will for all bug\/regression\/errors explain what the error is, where it is (include type names and namespaces when possible), why it's an error, and if possible short guidance on how to fix it. Do not use any type of bullet-list formatting\")]\r\n\u00a0 \u00a0 public string[]? PotentialIssuesFound { get; init; }\r\n\r\n [Description(\"A List of other repo files that could improve the review if you had them\")]\r\n\u00a0 \u00a0 public ReviewAdditionalFile[]? AdditionalFilesThatCouldImproveReview { get; init; }\r\n\r\n}<\/code><\/pre>\nWe now call the agents InvokeAsync method to get the Review object (and the token usage statistics).<\/p>\n
var chatHistory = new ChatHistory();\r\nchatHistory.AddUserMessage($\"\"\"\r\n Task: {taskDefinition.Title}\r\n - Description: {taskDefinition.Description}\r\n ---\r\n PR Diff: {prDiffContent}\r\n \"\"\");\r\n\r\nawait foreach (var content in agent.InvokeAsync(chatHistory))\r\n{\r\n\u00a0 \u00a0 var json = content.ToString();\r\n\u00a0 \u00a0 var tokenIn = 0;\r\n\u00a0 \u00a0 var tokenOut = 0;\r\n\u00a0 \u00a0 if (content.Metadata?.TryGetValue(\"Usage\", out var usage) == true && usage is ChatTokenUsage chatTokenUsage)\r\n {\r\n\u00a0 \u00a0 \u00a0 \u00a0 tokenIn = chatTokenUsage.InputTokenCount;\r\n\u00a0 \u00a0 \u00a0 \u00a0 tokenOut = chatTokenUsage.OutputTokenCount;\r\n }\r\n\u00a0 \u00a0 var review = JsonSerializer.Deserialize<Review>(json)!;\r\n}<\/code><\/pre>\nSome might think, why not ask AI what extra files it needs prior to the Review? It is done this way in order to save Input Tokens which is by far the biggest cost here. So by levering the excellent Structure Output feature we only need to give the PR-diff twice if additional files be needed, and if not a single pass to the LLM was all we needed.<\/p>\n
Step 6: Add Pull Request Comment<\/strong><\/h2>\nAs what you now do with the Review is a matter of opinion. We have chosen to add a simple PR Comment with potential comments to keep it short, and with a link to the longer review displayed at a link to the full review on an Internal Website.<\/p>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/03\/prComment.png\")
<\/a><\/p>\n… But it is also possible to instruct the AI via the structured output to give line-by-line Comments if that suits your team (it is clever enough to do it) or post the full review directly as a GitHub comment.<\/p>\n
The review is now done; If no findings, then it is time for a human to make the final review, but if anything is found the original developer can immediately react to it and not lose flow waiting for colleagues to have time to do the review.<\/p>\n
Reception from the developers<\/strong><\/h2>\nAs we started a prototype of this project prior to having API Access to the Open AI reasoning models, early iterations of the Agent sometimes produced inaccurate findings, leading to skepticism among the developers. However, after switching to reasoning models and refining instructions, accuracy has significantly improved. It has even become a thing for one of the developers to do this \ud83d\ude09<\/p>\n
Below are the building blocks and the flow of the AI Agent (each of the 6 steps is explained in detail below)<\/p>\n
When the Developer creates their Pull Request and marks it as ready a GitHub Webhook triggers and informs the AI Agent that it is time to do a review.<\/p>\n The AI Agent now determines, based on its setting what type of Review it should do and gathers the PR Diff Content (what code the developer has added, changed, and removed)<\/p>\n Here is how we retrieve the PR diff using the GitHub API<\/em><\/p>\n Beyond the Code, we also want to feed the AI with a title and description of what task, linked to the PR, that the developer set out to achieve, in order for the AI to evaluate if the code actually accomplishes the job.<\/p>\n We now have all the data needed for the review, we create our Semantic Kernel object with o3-mini-high for Azure Open AI Services as backend.<\/p>\n Our Agent<\/p>\n Note that we use o3-Mini’\u00a0ReasoningEffortLevel and the\u00a0ResponseFormat\u00a0is set so instead of a normal response we get\u00a0structured output<\/a>\u00a0that reduce the Developer (System) Message and allow us to get both the review and additional requested files back in a single request.<\/p>\n The Structured Output class is as follows.<\/em><\/p>\n We now call the agents InvokeAsync method to get the Review object (and the token usage statistics).<\/p>\n Some might think, why not ask AI what extra files it needs prior to the Review? It is done this way in order to save Input Tokens which is by far the biggest cost here. So by levering the excellent Structure Output feature we only need to give the PR-diff twice if additional files be needed, and if not a single pass to the LLM was all we needed.<\/p>\n As what you now do with the Review is a matter of opinion. We have chosen to add a simple PR Comment with potential comments to keep it short, and with a link to the longer review displayed at a link to the full review on an Internal Website.<\/p>\n … But it is also possible to instruct the AI via the structured output to give line-by-line Comments if that suits your team (it is clever enough to do it) or post the full review directly as a GitHub comment.<\/p>\n The review is now done; If no findings, then it is time for a human to make the final review, but if anything is found the original developer can immediately react to it and not lose flow waiting for colleagues to have time to do the review.<\/p>\n As we started a prototype of this project prior to having API Access to the Open AI reasoning models, early iterations of the Agent sometimes produced inaccurate findings, leading to skepticism among the developers. However, after switching to reasoning models and refining instructions, accuracy has significantly improved. It has even become a thing for one of the developers to do this \ud83d\ude09<\/p>\n<\/a><\/p>\nStep 1: Developer creates Pull Request<\/strong><\/h2>\n
Step 2: Get the Pull Request<\/strong><\/h2>\n
public async Task<string> GetPrDiff(GitHubClient client, string owner, string repo, int pullRequestNumber)\r\n{\r\n\u00a0 \u00a0 var diffUrl = $\"https:\/\/api.github.com\/repos\/{owner}\/{repo}\/pulls\/{pullRequestNumber}\";\r\n\u00a0 \u00a0 var diff = await client.Connection.Get<string>(\r\n\u00a0 \u00a0 \u00a0 \u00a0 new Uri(diffUrl),\r\n\u00a0 \u00a0 \u00a0 \u00a0 new Dictionary<string, string>(),\r\n\u00a0 \u00a0 \u00a0 \u00a0 \"application\/vnd.github.v3.diff\"\r\n );\r\n\r\n\u00a0 \u00a0 return diff.Body;\r\n}<\/code><\/pre>\nStep 3: Get Task Definition<\/strong><\/h2>\n
Step 4: Conduct the AI Review (+ optional Step 5)<\/strong><\/h2>\n
var kernelBuilder = Kernel.CreateBuilder();\r\n\r\nvar httpClient = new HttpClient\r\n\r\n{\r\n\r\nTimeout = TimeSpan.FromMinutes(5)\r\n\r\n};\r\n\r\nkernelBuilder.AddAzureOpenAIChatCompletion(\"o3-mini\", endpoint, apiKey, httpClient: httpClient);\r\n\r\nvar kernel = kernelBuilder.Build();<\/code><\/pre>\nvar agent = new ChatCompletionAgent\r\n\r\n{\r\n\r\nName = \"RelewisePrReviewer\",\r\n\r\nKernel = kernel,\r\n\r\nInstructions = \"\"\"\r\n\r\nYou are a highly experienced and extremely thorough C# code reviewer.\r\n\r\nYour task is to review the provided GitHub PR diff.\r\n\r\nYou will particularly pay attention to logical and performance bugs and\r\n\r\nregressions, and generally be very meticulous in determining what this PR\r\n\r\nintroduces in terms of new and\/or changed features and\/or behaviors.\r\n\r\nShould you at any point discover that it would be beneficial to you as the\r\n\r\nreviewer to see additional types, you may at any point continue to ask me for\r\n\r\nadditional types, and then I may provide them to you.\r\n\r\n\"\"\",\r\n\r\nArguments = new KernelArguments(new AzureOpenAIPromptExecutionSettings\r\n\r\n{\r\n\r\nReasoningEffort = ChatReasoningEffortLevel.High,\r\n\r\nResponseFormat = typeof(Review)\r\n\r\n})\r\n\r\n};<\/code><\/pre>\npublic class Review\r\n{\r\n [Description(\"A Short, concise but sufficient summary of what this PR introduces\/changes, and any general observations you find relevant in the context of the review, focus on functional changes that make a difference to the users of the system\")]\r\n\u00a0 \u00a0 public required string Summary { get; init; }\r\n\r\n [Description(\"You will for all bug\/regression\/errors explain what the error is, where it is (include type names and namespaces when possible), why it's an error, and if possible short guidance on how to fix it. Do not use any type of bullet-list formatting\")]\r\n\u00a0 \u00a0 public string[]? PotentialIssuesFound { get; init; }\r\n\r\n [Description(\"A List of other repo files that could improve the review if you had them\")]\r\n\u00a0 \u00a0 public ReviewAdditionalFile[]? AdditionalFilesThatCouldImproveReview { get; init; }\r\n\r\n}<\/code><\/pre>\nvar chatHistory = new ChatHistory();\r\nchatHistory.AddUserMessage($\"\"\"\r\n Task: {taskDefinition.Title}\r\n - Description: {taskDefinition.Description}\r\n ---\r\n PR Diff: {prDiffContent}\r\n \"\"\");\r\n\r\nawait foreach (var content in agent.InvokeAsync(chatHistory))\r\n{\r\n\u00a0 \u00a0 var json = content.ToString();\r\n\u00a0 \u00a0 var tokenIn = 0;\r\n\u00a0 \u00a0 var tokenOut = 0;\r\n\u00a0 \u00a0 if (content.Metadata?.TryGetValue(\"Usage\", out var usage) == true && usage is ChatTokenUsage chatTokenUsage)\r\n {\r\n\u00a0 \u00a0 \u00a0 \u00a0 tokenIn = chatTokenUsage.InputTokenCount;\r\n\u00a0 \u00a0 \u00a0 \u00a0 tokenOut = chatTokenUsage.OutputTokenCount;\r\n }\r\n\u00a0 \u00a0 var review = JsonSerializer.Deserialize<Review>(json)!;\r\n}<\/code><\/pre>\nStep 6: Add Pull Request Comment<\/strong><\/h2>\n
<\/a><\/p>\nReception from the developers<\/strong><\/h2>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/03\/thanksAi.png\")
<\/a><\/p>\n