At Microsoft, we are constantly exploring innovative ways to enhance our workflows, improve productivity, and ensure we have a secure supply chain that uses modern tools. One of our recent initiatives was to rebuild the .NET Framework build system by leveraging AI to automate the process. Here’s the story of how we successfully achieved this transformation.

The Problem

Modernizing the .NET Framework build system required converting about 300 projects from custom build files to MSBuild, including renaming properties and updating imports. We knew this process would be both time-consuming and error prone. While some conversions were more complex than others, we knew many followed similar patterns, which presented an opportunity for automation.

The Experiment and Solution

Consequently, we explored leveraging AI to streamline these project file conversions. AI could identify recurring patterns, apply necessary modifications, improve accuracy, and reduce manual effort while accelerating the overall modernization process. Given the potential benefits, we conducted a weeklong experiment to evaluate AI’s effectiveness in automating these conversions and validating our approach.

We decided to experiment with two different methods of converting projects:

Copilot Chat Participant: We developed a VS Code extension that provided a Copilot Chat participant to assist in project conversions. The participant took a chat message requesting Copilot to convert a single project file and iterated with Copilot until the project successfully built. The extension added context to the prompt with manually written documentation that included details about MSBuild, the task of converting an MSBuild project, and patterns from previous conversions. It also offered Copilot a selection of tools that it could invoke. These tools provided the ability to edit files, gather additional information about the MSBuild project file and workspace, and run MSBuild. The participant would execute tools that Copilot requested and provide the results in a loop until Copilot was satisfied with the contents of the project.
Script-based Documentation and Conversions: We developed a Python script that utilizes Azure OpenAI’s GPT-4o-mini and o1-mini APIs with the following functionalities:
- Documentation Generation: The script generated comprehensive documentation that outlined the rules for conversion, including the XML structure before and after the conversion, as well as optional pseudo-code. The documentation was generated by getting the o1-mini model to infer the differences between the old .csproj and manually generated ones and then synthesize rules. A benefit of getting the language model to generate the documentation was to help other engineers onboard quicker.
- Project Conversion: Based on the generated documentation, the script applied the conversion rules to a new project. It then ran MSBuild on the converted project to verify its successful execution. To save time, we asked the LLM to generate 3 different versions of the converted projects and chose the first one that was successfully built using MSBuild; this strategy reduced the overall conversion time by considering the probabilistic nature of Language Models where each new version of the converted project was usually different from the previous one.

Results

Both implementations yielded promising results converting project files. When tested on files that had already been converted (undoing the manual conversion and converting it with AI), particularly files under 200 lines of code, they performed well and produced projects that built, and often even made important fixes that were missing from manual conversions. However, occasionally, a one-shot conversion often missed key details like renaming properties or updating imports correctly. However, this was still a great starting point for an engineer to manually review and fix.

The Copilot Chat participant’s major drawback was that it worked on one project at a time and had slow generation. For basic projects, it ended up taking a similar amount of time to generate the project and review it compared to manually converting the file. For this reason, we decided to move forward with the script instead.

The script-based approach yielded similar results for each individual conversion but more importantly was able to convert multiple projects at once. While the AI still made mistakes and some manual reviews and cleanup were still necessary, it was much easier to convert several projects at once and review the ones that failed to build. This led to noticeable improvements in project conversion velocity.

After AI conversions were introduced, the project burndown speed increased drastically, and we were able to reach the ambitious goal well ahead of schedule. We were able to convert simple projects quickly and easily with AI, while manually converting complex and unique projects.

Cost

Running experiments and conversions using Azure OpenAI’s GPT-4o mini and o1-mini models cost less than $1; this was another win as apart from the speed at which we converted projects, the language model usage wasn’t cost prohibitive.

Conclusion

We dedicated two weeks of a couple of engineers’ time to experiment using LLMs to automate converting MSBuild projects to the new build process and had great success. We encountered some issues with MSBuild’s verbosity and error messages when implementing agentic behavior, and the LLMs occasionally generated imperfect projects that required manual intervention. Despite this, utilizing AI increased our velocity significantly and enabled us to complete our goal ahead of an ambitious timeline.

Since this project, we’ve learned of other possible solutions that we could have explored. If we were to do this again, we would make use of a custom Model Context Protocol server in VS Code such as described here to invoke the conversion process and give the model even more context for the conversions.