When we first set out to get smarter AI edits in Visual Studio Copilot, we knew we were tackling a deeply complex problem. It wasn’t just about generating great suggestions—it was about figuring out how to seamlessly apply those suggestions to your code. While the idea seemed simple at first glance, the reality was anything but.
The Complexity of Implementing AI-Generated Edits
Let’s rewind to our early attempts. Copilot might give you a fantastic suggestion—a new method, a helpful refactor, or maybe even a corrected logic block. But the real challenge began when we tried to insert those changes into the existing file without breaking anything. Where does the edit go? What happens if the file has been updated since the suggestion was generated? What if the model’s output inadvertently introduces conflicts, overlaps existing code, or even forgets a required bracket? These questions made the process tricky, and the resulting red squiggles made it difficult to trust AI suggestions.
In the early days, we approached this problem in the most straightforward way possible: brute force. We used heuristics and rule-based techniques like string matching and pattern recognition to identify where edits might belong. It worked… kind of. The results were often inconsistent, especially for complex edits spanning multiple lines. And as Copilot’s capabilities expanded to support more languages and scenarios, maintaining these rules became a moving target—always shifting, always growing harder to manage. Success rates hovered around 50%, which was far from ideal. In hindsight, it became increasingly clear that we couldn’t outpace the ever-evolving models with static rules.
Better Models Meet Smarter Techniques
As AI technology advanced, we saw an opportunity to revisit this problem with a fresh perspective. Two major developments in AI made it possible to rethink how we could get smarter edits in Visual Studio Copilot: modern models with larger context windows and a groundbreaking technique called speculative decoding.
Speculative decoding became a game changer for speeding up AI-assisted edits. It works by pairing a fast model with a more sophisticated one. The fast model generates token predictions for the edit ahead of time, while the smarter model steps in only when needed to refine or verify those predictions. This collaboration improved average token generation speed by 2-3x and made it feasible to implement the generated edits using the model, which significantly enhanced accuracy when integrating changes into your files.
By using speculative decoding, we adopted the model-based approach to applying edits that was able to fill in the gaps that the previous approach could not reach. Instead of relying on rule sets, we employed an AI model to simulate an “ideal” version of your file with the chosen suggestions seamlessly integrated. A smart diff algorithm then compares this ideal version with your actual file to pinpoint and precisely map the edits. This allowed the entire process to handle edge cases—like overlapping code or missing syntax—more intelligently than ever before.
Balancing Accuracy and Speed
While speculative decoding dramatically improved performance, applying AI edits still came with a natural trade-off: using models introduced latency. Previously, all string computation was done locally, meaning edits appeared almost instantly. Now, every edit involved network calls and token generation—a process that unfolds incrementally as a token stream, rather than as a single, instant response.
To make this experience feel meaningful for users, we implemented a streaming animation in the editor. As edits are detected on the token stream, users see their document updating line by line in real time. This animation serves two purposes: showcasing progress and providing visibility into exactly how the changes are being applied. Although this approach trades the speed we had before for greater accuracy, feedback has shown that users value precision over raw speed—especially when it comes to maintaining code quality and reducing disruption.
Looking Ahead
This work laid the foundation for Agent Mode in Visual Studio Copilot—a feature that enables the AI to not only suggest improvements but actively assist in executing them. For Agent Mode to function effectively, the edits needed to be precise, seamless, and reliable enough for the agent to build, debug, and test the code. Speculative decoding has already made significant strides in balancing speed and accuracy, but we’re not stopping there.
A faster implementation of our speculative decoding technique is on the horizon. This updated version will leverage advancements in token generation and model pairing to further reduce latency while maintaining accuracy users have come to expect. Early tests show promise of a 2-4x speed boost compared to the original rollout, moving closer to the ideal experience where precision meets near-instant response times.
How’s the progress on or plan for large files? Our codebase is quite old and we have lots of files that are thousands of lines of code and applying code changes seems to go through the entire file from Line 1 to the end even if there is only one particular method being changed. It takes forever and isn’t really usable. Not to mention that agent mode typically runs out of context window anyway, making the entire process fail.
I have always been worried when allowing AI to add code – it never goes to the right place.
I am not sure I trust things enough to even want to deal with messy code.
Trust is easily broken.
It’s definitely getting better, using Copilot now for almost 2 years already.
The diff algorithm is improved, sometimes it still misses a one off closing bracket when inserting code.
Currently what mostly annoys me is when Copilot makes changes to code that are not at all related to the task, like removing comments, updating method descriptions in XML comments, reformatting sections unrelated to the code changes, and moat of all not obeying the static instructions from the .MD file.
Can’t wait until you can have Copilot instruct your engineers to fix the broken checkbox in your application, which has been broken for over 421 days now. #FixTheCheckbox https://developercommunity.visualstudio.com/t/Closing-Chrome-Browser-Does-Not-Close-De/10678107
Broken? It works fine on my machine across Firefox, Chrome, Edge and Edge Beta. Perhaps this is why they are ignoring you as it isn’t broken.
Thank you for your feedback. If you read the issue, you'd see that this problem was confirmed by Visual Studio staff a year ago. Additionally, Firefox works, so the fact you mention this suggests you do not understand the situation. Both Chrome and Edge browsers fail to close the debugger when the browser closes, despite the checkbox in options indicating that it should. Note that you must be running as Administrator and Script Debugging must be enabled. If you can confirm that this doesn't fail for you, we'd love to have you on the ticket...