Simplified Code Refinement and Debugging with GitHub Copilot Chat
This March, we announced GitHub Copilot chat for Visual Studio 2022. With Chat, Copilot has moved beyond code completions, offering in-depth analysis and explanations of how code works. It supports the generation of unit tests, and even helps with proposed fixes to bugs, and explaining exceptions. By gathering useful context data from Visual Studio, Copilot Chat helps you form great questions to get useful answers. Using GitHub Copilot with Visual Studio gives you more time for creativity by spending less time on boilerplate manual tasks and diagnosis.
Now, we’ve taken deep integration with Visual Studio a step further. Using the Interactive Code Assistant view, you can seamlessly refine your code with Copilot Chat, without ever needing to leave the comfort of your editor window. Your workflow is uninterrupted – no more going back and forth to the chat window! Get started using “ask Copilot” wherever in your code file you want to work – ask your questions and see answers inline with the code they relate to. Whenever Copilot suggests code changes, you’ll see your code side by side with Copilot’s suggestion – the normal Visual Studio diff view pattern – to help you thoroughly review, correct and refine what is being proposed at your own pace before applying to your code. We’ve taken care to make sure you are always in the driver’s seat, with complete control throughout the process.
Check out the video below to see how Copilot’s Interactive Code Assistant works:
Enhanced diagnostics with Copilot
The team’s also been hard at work bringing Copilot Chat to your fingertips in more places when you are debugging in Visual Studio. The three features below illustrate some of that work:
CPU Usage auto insights in the profiler
A profiler can help you make informed decisions quickly by providing a visual depiction of execution times and CPU usage for your application. Copilot now gives you detailed information via CPU Usage auto insights. It now provides more detailed information and insights for specific methods and properties, including Enum.HasFlag, Enum.ToString, String.StartsWith, ConcurrentDictionary.Count and more. With Copilot, you can ask questions about functions on the identified hot paths in your code, which can help you produce more efficient or cost-effective code. Just click “Ask Copilot” to start exploring.
Copilot Exception Helper
Copilot can provide you with meaningful help when you are debugging your code. For example, when an exception is thrown, it gives you the opportunity to start asking questions. Copilot has access to exceptions, call stack, local variables, and code. By forming good questions based on the right parts of the data Visual Studio has when you’re at an exception, Copilot Chat can provide useful insights and fixes for the issue.
See how this works in this video
Debugger “Rubber Ducking”
Copilot can now understand call stacks, frames, variable names, and values. This means you can pose detailed questions to a debugger-aware Copilot.
Check out the video below to see how this works:
How can I get these Copilot Chat preview features?
All Copilot for Individuals users now have access to the GitHub Copilot Chat preview for Visual Studio! See Shuyin Zhao’s blog post for more details. If you are already a user of Copilot for Individuals, just get the latest version of the extension to see the new features.
We want to hear from you!
We’re really grateful for all the feedback you’ve provided so far as we’ve been creating Copilot Chat, and excited to see where this next wave of AI-assisted productivity takes you as you create great software. Please try the new features and share your feedback – just use Visual Studio “Report a Problem” to send us your suggestions, feedback and issues for GitHub Copilot Chat in Visual Studio.
Mark Wilson-Thomas Principal Program Manager, Visual Studio Copilot Chat
Light
Dark
16 comments
I tried Copilot a few months ago and was not impressed. The AI drifts too much, I said “don’t use types, use ‘var'”. And the AI changed. Then I said refactor this and that. And boom, types were back. I said “make all local variable names lowercased pascal”. And the AI did that. Then I asked for another refactor and boom the new variables where uppercased pascal. 1 step forward, 2 steps back. When ever I instruct something, the AI breaks something else.
Simple examples work and feel like magic, but the moment it gets a bit more complex, it can get frustrating very quickly. Also the AI is over-trained with the same primitive examples of Northwind database, a pizza delivery system with a shopping cart, and a people management system with a person class with name, first name, age etc. It “spills” into everything that has nothing to do with any of the 3.
Also, the text to write/instruct is way too long as are the answers, to use it in a productive way. “root1 is Nan, why?” — “parameter of Math.Sqrt() was negative” — “oh, fix that with a catch-if” — “done”. Easy, simple, quick. I use ChatGPT4 a lot and I think the text I write the most is “short answers, please”. I know you all mean well but you have to be careful not to explain to developers with a masters degree the difference between a mouse and a keyboard.
Hi Melissa
Thanks for taking the time to try Copilot Chat, and for your feedback.
We are just getting started with Copilot Chat, and will be working on improvements to make the system more useful over time. Whilst it does have limitations, we believe it can be useful for many cases. Thanks for giving us a sense of your priorities with this post; I’m hearing you’d appreciate the following features:
1. If you have (or express) a convention in your queries, you’d like a way to make that “stick” for future queries on that codebase. E.G if you prefer var to types, you might prefer to have it honor your stated preference in your questions, or perhaps even from your .editorconfig file if supported?
2. You would appreciate being able to ground Copilot’s responses in more complex cases (perhaps even your own company/org’s repos?)
3. You’d prefer a more terse form of answers in explanations. You can already control this to some degree by asking for it in your question (give me a terse description of ….) but it sounds like you’d like a way to control it globally as a personalization of the responses?
I’m happy to carry on this conversation on email too – my address is mwthomas at microsoft dot com,
Thanks
Mark
I nowadays mostly use GPT4, but to my knowledge Copilot is using that in the background. With GPT I usually start a session and give instructions, which I now can pre-define. Short answers, reduced comments and compact/condensed code. And I tell GPT if I currently work using C#, C++ or HLSL. It works quite well but still from time to time I have to remind GPT to get back to my settings.
I think for me personally, the most important thing is consistency above all. Every company defines its own rules of code writing that everyone has to stick to, so GPT needs to follow these rules strictly as well. As the conversation is verbal, it should be verbal was well to give these instructions, but I dont think that it really matters. It should be a one-time thing to say it has to be this or that way and GPT needs to stick to it.
And if I may make some comments about the last video with the quadratic roots: The answer is so long, it doesn’t even fit the screen. I would have to scroll up and down to read everything. And most of the text is written like conjecture and “it may be wrong” and “please don’t sue if it’s wrong”. I understand the reason why now everything needs a mile long disclaimer but I just don’t want to read them again and again. And then finally the suggestion “can you explain what a quadratic equation is?”. It makes me feel bad, as if I don’t know what I’m doing and just stumble upon some code I don’t understand. That’s not what I want. I want GPT to consider me as someone who knows all about what’s going on and just have some brain freeze and don’t see what the problem is and GPT says “you forgot about the negative determinant” and I say “omg, of course, how did I miss that obvious mistake” because that’s what it is, or should be. The level of professionalism is not equal, if that makes sense. I want Copilot to support me, not to train me. I have always the option to ask further questions. To get more detail. And I want to have this more reflected in the AI responses. “can you tell me about all special cases for quadratic equations”…. “can you write the root formula for quad equations” .. “can you verify the equation i use is correct” “can you give some numeric examples to test my code”…. these are so much better question templates/links than “what is math”, “what is an equation” … I dont know if I make sense here, I just want to be “treated” by the AI as a professional and not a first time student.
And don’t get me wrong. We all started from scratch some day ago and had to get a Hello World running. But most users will have moved on. And just don’t see what’s wrong with some current code or think about the issue at hand. Why the code isn’t working.
Today I asked GPT “what windows API tells the system to not go to sleep?” — I used that function a hundred times, it has a weird name, I never remember it. “The Windows API function SetThreadExecutionState is used to inform the system not to enter sleep mode or screen saver mode.” was the reply. Short, precise, no fluff. It saved so much time, it is so helpful. And I think that’s the case for most of us. We all know exactly what we want to do, we just lack that little detail, the name, the parameter.
Last week I needed to check if a line intersects a rectangle, I’ve done that a hundred times, it needs like 8 checks but it’s hard to remember all of them. I was lazy and didn’t wanna bother with it, so I asked GPT to write the code for me. I didn’t need a mile long explanation what it does, I know exactly what it does or how it works; I was just lazy and why shouldn’t I, I’ve done it many times but didn’t want to waste time on searching or copying my old code.
It’s a long text, but these are the things I want from Copilot. I wrote GPT but it’s the same. Support not teaching. When something is unclear I can always ask for more information.
Thanks so much for the extra detail. It helps us understand where to invest next.
Mark
Totally agree here, been using Copilot since it was available, both private and at work. Seeing pretty much the same issues. Where I find it most useful is actually filling in the blanks, like completing tests, inserting logical or calculated values in tests, also in writing comments and documentation, it pretty nicely completes the info.
As a debugging helper I really couldn’t get it to do anything remotely useful. Exception explanations may possibly be useful for newbies, but for anything a bit more complex its returning total nonsense or nothing more than what the exception already reveals.
I did use it with success in VSCode to write a working angular app (load an image, set/unset points on it, record the location x/y, export later as json in a given format), I have not the slightest idea about angular, yet i got it to work nicely with Copilot.
But having predefined settings would be really useful for Copilot chat. When looking at its debug output one can see that it does lots of context definitions (like “you are a professor teaching software development…” funny stuff), adding user settings should be easy.
Addendum: for enterprise use it needs more context, like on the solution level, it must be able to understand the WHOLE solution code not only the opened files, or whatever is its current context level. Most often Copilot does not understand how some of the classes used actually work, and it does not suggest them in new code even if previous code did use them.
Thanks for this feedback Andreas. We are definitely aware of this requirement for whole solution context. If you can share some cases that would be particularly important to your usage in this regard that would be great and help the team prioritize the most useful cases. It sounds like you’re looking for understanding/explanation/diagnosis that spans multiple files in the solution as your main case?
Also noting
1. Your comments about increasing debugger help usefulness – the team has added more context there in recent releases, as Mark Downie noted in the debugging section, so please do give it another go, and raise feedback tickets about cases where it doesn’t help you.
2. Your support of some level of personalization of the verbosity and style of output. Out of interest would you also pick a very terse style?
Thanks
Mark
Mark, for what its worth, the most use of AI in debugging scenarios would probably come from having it analize the call stack and variable flow, not only the exception context. But this is probably a bit off the capabilities of the current implementation.
I definitely would appreciate being able to set an “experience level” lets say, so Copilot will reduce redundant information and present a appropriately terse response. Add “explain more” or add links to items that can add more information to the context, but dont show it automatically in an overly chatty response.
Something like “I am a C# .NET dev with 10+ years experience”, don’t tell me what the Exception class is used for, tell me why this exception is thrown and why the variable is NULL at this point.
I am sure that the debug helper is more than useful for beginners and even for developers in a junior position. For an experienced developer is mostly redundant now.
As for the “Solution Context”, yes, I am definitely looking for something that can tell me that the values in a given class are the result of an EF database query. That is able to create for example a dataset by using classes that already exist, with a instruction like “return a dataset that contains all records from the XXX table filtered by the xyz parameter for the date field and ordered by the Z index, create a class that holds the data for this dataset and can be exported as JSON”
And it will create the class needed, with properties to access the data, JSON attributes for them, uses the existing EF classes to query the data. Asking too much? Possibly.
I’ve been using the preview Copilot / Copilot Chat since Build and unfortunately it has been a massive disappointment. It doesn’t work anything like it does in Microsoft’s demos.
– The in-context Copilot features just don’t work at all. I almost never get code suggestions and it doesn’t attempt to suggest code based on comments. Even when it does offer suggestions, its simple stuff like a semicolon. I’ve never had it offer to complete an entire method as is shown in the demos.
– Copilot Chat is frankly awful. It is incapable of analyzing even the most simple of methods in my experience, and rather than tell you that, it just completely lies and makes up details about the method that are not true. I asked it to tell me the difference between two similar methods and its explanation was literally 100% BS that had nothing to do with what the methods actually did.
– 70% of the code that it generates is completely wrong and useless. I’m assuming what is happening is that in my large solution, VS is passing too much context and the GPT is choking and dying on it.
– After 5-6 prompts with Copilot Chat is goes berserk, stops answering prompts in sentences and starts responding with largely irrelevant nonsense.
I’m guessing that, as usual, Microsoft is only testing with toy “todo list” apps rather than anything resembling a real-world solution with many projects. I’ve found it to be substantially inferior to just asking ChatGPT-4 targeted questions.
Hi MgSam
We have been updating Visual Studio Copilot and have made a number of changes since Build in the preview bits. Based on testing and feedback we have a number of improvements planned in the area of context management too. The new inline chat feature makes context easier to control too – I encourage you to try it.
We are still in the preview phase, and would very much welcome your bug reports from Visual Studio via the “report a problem” feature, including the Copilot Chat logs from your Output Window’s “Copilot Chat” pane – if you have cases like the ones mentioned above please raise feedback tickets and we’ll dig in to understand and fix.
Thanks
Mark
I’m liking Copilot a lot, I’ve had it in Visual Studio for about 5 weeks now and I didn’t really use at the beginning. I started using it by replacing my searches on google and stack overflow for syntax or finding the best way to do something.
The suggested code has saved me probably thousands of key taps a day! You know when you start writing a class and like intellisense, it suggests properties given the name of the class.
My next project is going to have a lot of handler classes… I wonder if I could write a couple of them, keep the files open in VS and then ask Copilot to create more for the crud operations on the other entities. That could be interesting.
Looking forward to see how Copilot evolves over time. 🙂
I’m glad to hear Copilot is helping you so far! Would love to hear how your experiment goes; we are looking at further improvements to context going forward and your case is an interesting one. Please do share – you can drop me a line anytime mwthomas at Microsoft dot com or raise suggestions and feedback via “report a problem” in Visual Studio.
Let us know what you think of the new inline refinement experience?
Regards
Mark
I assume this can be disabled because God knows there’s nothing I want more than a half-baked back seat driver looking over my shoulder and making questionable suggestions on how to code better and triggering my ADHD.
Guys, maybe fix the problems VS actually has before gluing more stuff literally NO ONE asked for into it?
But I DO have to say THANK YOU for putting a feature to move the menu bar out of the title bar. Been begging for that for so long…
Hey there!
This feature is currently in preview and requires an explicit sign-up step to get involved, so you certainly do not have to use it.
While I do not imagine that all developers will find this equally helpful, I am seeing a lot of early evidence that some developers are already relying on conversational assistance while coding. We just want to make sure we have the right tools available for the many kinds of developers we encounter.
The goal is to fix bugs and respond to community requests while also delivering new features for the broadest variety of Visual Studio developers.
Thanks for the feedback on the menu bar 🙂
Mark D
Is this available for any version of Visual Studio? Becuase I need to generate .NET 3.5 code all the time, and my most comfortable environment is Visual Studio 2008 for this. I use ChatGPT V2, occasionally for this, more for a sounding board to get ideas when I’m kinda stuck on how to approach something, and ChatGPT keeps forgetting that I specified .NET 3.5 coding, and outputs uncompileable solutions that I have to figure out. This would be less of an issue if someone wasn’t making it a point of erasing all the old VS2008 and .NET 3.5 help-pages and content from the internet. And I don’t have a budget for using a paid AI-service of any kind.
Hi Steve
The GitHub Copilot Chat Preview is currently only available for Visual Studio 2022 17.6 and higher, and as well as being admitted to the preview you’ll need a subscription to the GitHub Copilot service (see GitHub’s documentation) to use it.
Regarding help docs: whilst Visual Studio 2012 and earlier are out of support, you can still see docs in the following locations:
1. Visual Studio 2008 docs are on /previous-versions here
2. .NET 3.5 docs are here
I hope that helps
Mark