Coding with AI

Coding with AI

AI tools have quickly improved and our team made a purposeful effort to use them during our last engagement. This post will briefly talk about the tools we used and the results we got. We hope this post gets you excited about how you can use AI tools in your own work.

GitHub Copilot

GitHub Copilot was the obvious place for our team to start. Most of the team uses Visual Studio Code and the remainder use Visual Studio. Copilot is an extension that can be installed in both of these IDEs (as well as others). It gives code completion hints in the same way that Intellisense might, except the scope of what it can offer is much greater. It can write functional code based on understanding the context of the code you are writing.

While Copilot can write code for you, we didn’t find that to be the most compelling case. When Copilot writes code, it generally gives you something useful, but only about half the time it is what the developer was trying to write. However, there were many other exceptional use-cases:

• Write Documentation. We found it especially useful to document functions after they were written. In most cases, it could write a useful description of the function and parameters that needed very little or no input from the developer.
• Write Unit Tests. Given that setup and assertions are often consistent across unit tests, Copilot tends to be very accurate at anticipating the code you need in a new unit test. This means generally after creating a new test function, you can “tab-through” most of the implementation and then change what you need to for the test.

  Notice in the below example how it was able to write a fully functional new unit test based on other unit tests in the same file.

  

• Unblock. It is often hard start writing when staring at a blank page, Copilot can fill the space with something that may or may not be what you ultimately want to do, but it can help get you in the right mindset.

TIP: If you want Copilot to write something useful for you, try writing a comment that describes what your code is going to do – it can often take it from there.

You can find out more information about GitHub Copilot here.

ChatGPT / Bing Chat

For writing new blocks of code, our team found good results using ChatGPT and Bing Chat. Despite Bing Chat being able to query the most up-to-date information on the web and using GPT4 (instead of GPT3.5), we generally found ChatGPT produced better results. There isn’t a wide gap though, so we would recommend trying both. We did get much better results with Bing Chat in Creative Mode.

Most of the code generated this way was not based on prompts from the code we were working on, but rather from prompts to write specific functionality. For example, we might ask ChatGPT to “write a C# function that enqueues a message to an Azure Service Bus”. We will get back some code and then maybe further tweak it with additional prompts until we get something that includes all of our functional requirements (retry, log on error, etc.). We could then copy/paste that code into our code base and modify it as appropriate.

Here are some tips that might improve your usage of Large Language Models like this:

• Be Specific. A generic prompt like “write a C# function that enqueues a message to an Azure Service Bus”, doesn’t specify what version of C# or Azure SDK might be used, whether it should be async or not, etc. If you are specific, you can get back code that is more likely to be what you want. For example, “write an async C# function that enqueues a message to an Azure Service Bus using the latest SDK”.
• Challenge the Answers. If you see something that doesn’t look right or simply won’t compile, challenge it. You can tell ChatGPT specifically what didn’t work about the code. You can ask for citations on why it thinks this should work. You can challenge content in those citations. ChatGPT is capable of looking deeper into an issue and changing it’s answer if you give it direction.
• Change Perspective. ChatGPT can impersonate a persona or even a system and answer questions from that perspective. For example, you can ask it to explain what a particular piece of code does from the perspective of a user. You might have ChatGPT imagine it is a database administrator and ask it to explain how to improve a particular query. Simply asking ChatGPT to take on a different perspective can have a dramatic impact on what is produced.

  Notice in the below example we can set ChatGPT in a role and then ask follow-up questions from that perspective.

  

• Prompt Engineering. Chat AI tools are only as good as the prompts you give them. The quality and appropriateness of the output can vary greatly depending on the prompt. In addition, many of these tools restrict the number of prompts you can send in a given amount of time. To learn more about prompt engineering, you might review some open source documentation here.

GitHub Copilot X

Towards the end of the project, some of us got access to GitHub Copilot X features, most notably Copilot Chat. In most cases, this takes the place of using ChatGPT or Bing Chat. We had good results with this, on par with ChatGPT. It is restricted to “questions about coding”, which it can interpret narrowly, so there are still some cases for other tools. Beyond using the chat to generate code, we got good results asking it to do these things:

• Build SQL Indexes. Given a query, Copilot Chat could generate a SQL index that will improve the performance of the query.

  Notice in the below example we can provide a query and get index recommendations.

  

• Write Regular Expressions. These are notoriously difficult to write, but Copilot Chat could generate them given some sample input and a description of what to extract.

  Notice in the below example how easy it was to provide requirements and have Copilot Chat generate a regular expression.

  

• Improve and Validate. When we were unsure of the implications of writing code a particular way, we could ask questions about it. For instance, we could ask if there is a way to write the code that is more performant or uses less memory. Once it gives you an opinion, we could ask it to provide documentation validating that assertion.

There are some additional features we see value in, but did not have much opportunity to use:

• Explain. Copilot Chat can explain what the code is doing in natural language.
• Debug. Copilot Chat can analyze your code and propose solutions to fix bugs.

Copilot Chat writes code in the chat window which can then be inserted into an existing or a new file. It also has an inline feature where it can mutate your code directly in the editor window. For example, you can use the inline feature to add documentation directly to your code.

Considerations

We did not paste code with any sensitivity (for example, business logic or commercial code) into ChatGPT or Bing Chat. Since those systems are allowed to train on that data in future iterations, we did not want to risk exposing any sensitive information. We did not have the same concerns with Copilot Chat, since it is not allowed to train on the data it receives. Read more about how GitHub Copilot handles your data and code here.