{"id":230355,"date":"2020-08-24T08:00:56","date_gmt":"2020-08-24T15:00:56","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/visualstudio\/?p=230355"},"modified":"2020-08-24T10:39:03","modified_gmt":"2020-08-24T17:39:03","slug":"the-making-of-intellicodes-first-deep-learning-model-a-research-journey","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/visualstudio\/the-making-of-intellicodes-first-deep-learning-model-a-research-journey\/","title":{"rendered":"The making of Visual Studio IntelliCode’s first deep learning model: a research journey"},"content":{"rendered":"

Introduction<\/strong><\/span><\/p>\n

Since the first <\/span>IntelliCode<\/span><\/a>\u00a0<\/span>code completion\u00a0<\/span>model<\/span>\u00a0<\/span>was shipped<\/span>\u00a0in Visual\u00a0<\/span>Studio and<\/span>\u00a0Visual Studio Code<\/span>\u00a0<\/span>in 2018<\/span>,\u00a0<\/span>it has become an essential\u00a0<\/span>coding assistant<\/span>\u00a0for\u00a0<\/span>millions of<\/span> developers <\/span>a<\/span>round<\/span>\u00a0<\/span>the\u00a0<\/span>world.<\/span>\u00a0In\u00a0<\/span>the\u00a0<\/span>past two year<\/span>s, we have been\u00a0<\/span>working tirelessly to enable IntelliCode for more programming languages\u00a0<\/span>and<\/span>,<\/span>\u00a0<\/span>in<\/span> the meantime<\/span>,<\/span>\u00a0<\/span>researching<\/span>\u00a0ways<\/span>\u00a0to<\/span>\u00a0improve<\/span>\u00a0the model precision and coverage<\/span>\u00a0<\/span>to deliver\u00a0<\/span>an even\u00a0<\/span>more satisfying\u00a0<\/span>user experience<\/span>.\u00a0<\/span>One of\u00a0<\/span>our\u00a0<\/span>major research effort<\/span>s<\/span>\u00a0was to bring the latest advancement<\/span>s<\/span>\u00a0in deep learning\u00a0<\/span>for\u00a0<\/span>natural language <\/span>modeli<\/span>ng<\/span>\u00a0<\/span>to\u00a0<\/span>programming language\u00a0<\/span>modeling.\u00a0<\/span>After\u00a0<\/span>leveraging technologies like\u00a0<\/span>Azure Machine Learning<\/span><\/a>\u00a0and\u00a0<\/span>ONNX Runtime<\/span><\/a>,<\/span>\u00a0<\/span>we\u00a0<\/span>have\u00a0<\/span>successfully<\/span>\u00a0<\/span>shipped\u00a0<\/span>the first<\/span>\u00a0<\/span>deep learning model for all the\u00a0<\/span>IntelliCode\u00a0<\/span>Python\u00a0<\/span>users in V<\/span>isual Studio\u00a0<\/span>Code.<\/span><\/p>\n

The Research Journey<\/strong><\/span><\/p>\n

The<\/span>\u00a0journey\u00a0<\/span>started\u00a0<\/span>with a research exploration\u00a0<\/span>in applying<\/span>\u00a0<\/span>language<\/span>\u00a0model<\/span>ing\u00a0<\/span>technique<\/span>s<\/span>\u00a0<\/span>in<\/span>\u00a0natural language processing\u00a0<\/span>for learning\u00a0<\/span>P<\/span>ython code.<\/span>\u00a0<\/span>We\u00a0<\/span>focused on the current IntelliCode member completion scenario<\/span>\u00a0<\/span>as shown in\u00a0<\/span>Figure 1 below<\/span>.\u00a0<\/span><\/p>\n

\"Image
Figure 1. Example of member completions powered by IntelliCode for Python in Visual Studio Code<\/figcaption><\/figure><\/p>\n

 <\/p>\n

The fundamental task is to find the most likely member of a type given a code snippet preceding the member invocation. In other words, given original code snippet C, the vocabulary V, and the set of all possible methods M \u2282 V, we would like to determine:<\/p>\n

\"Image<\/p>\n

To find that member, we need to build a model capable of predicting likelihood of the available members.<\/p>\n

Previous state-of-the-art recurrent neural network<\/a> (RNN) based approaches only leveraged the sequential nature of the source code, trying to transfer natural language methods without taking advantage of unique characteristics of programming language syntax and code semantics. The nature of the code completion problem made long short term memory<\/a> (LSTM) networks a promising candidate. During data preparation for model training, we leveraged partial abstract syntax tree (AST)s corresponding to code snippets containing member access expressions and module function invocations, aiming to capture semantics carried by distant code.<\/p>\n

Training deep neural networks is a computationally intensive task that requires high-performance computing clusters. We used a data-parallel distributed training framework Horovod<\/a> with Adam optimizer<\/a>, keeping a copy of an entire neural model on each worker, processing different mini-batches of the training dataset in parallel lockstep. We\u00a0<\/span><\/span>utilized<\/span><\/span>\u00a0<\/span><\/span>Azure\u00a0<\/span><\/span>Machine Learning<\/span><\/span><\/a>\u00a0<\/span><\/span>for<\/span><\/span>\u00a0<\/span><\/span>model<\/span><\/span>\u00a0<\/span><\/span>training<\/span><\/span>\u00a0<\/span><\/span>and<\/span><\/span>\u00a0<\/span><\/span>hyper<\/span><\/span>\u00a0<\/span><\/span>parameter<\/span><\/span>\u00a0<\/span><\/span>tuning <\/span><\/span>because its on-demand GPU cluster service made it easy to scale up our training as needed, and it helped to provision and manage clusters of VMs, schedule jobs, gather results, and handle failures. The Table 1 showed the model architectures we tried and their corresponding accuracy and model size.<\/p>\n

 <\/p>\n

\"Image
Table 1. The top 5 accuracy and associated model size for the different deep learning model architectures<\/figcaption><\/figure><\/p>\n

 <\/p>\n

We chose to productize with predicted embedding due to its smaller model size and 20% model accuracy improvement comparing to the previous production model during offline model evaluation; model size is critical to production deployability.<\/p>\n

The model architecture is shown in the Figure 2 below:<\/p>\n

\"Architecture
Figure 2. Architecture diagram illustrating IntelliCode’s deep LSTM model<\/figcaption><\/figure><\/p>\n

 <\/p>\n

To deploy the LSTM model into production, we had to improve the model inference speeds and memory footprint to meet edit-time code completion requirements. Our memory budget was about 50MB and we needed to keep the mean inference speed under 50 milliseconds. The IntelliCode LSTM model was trained with TensorFlow and we chose ONNX Runtime<\/a> for inferencing to get the best performance. ONNX Runtime works with popular deep learning frameworks and makes it easy to integrate into different serving environments by providing APIs covering a variety of languages including Python, C, C++, C#, Java, and JavaScript \u2013 we used the .NET Core compatible C# APIs to integrate into the Microsoft Python Language Se<\/span><\/span>rver<\/span><\/span><\/span><\/a>.<\/p>\n

Quantization is an effective approach for model size reduction and performance acceleration if the accuracy drop introduced by low bit width numbers approximation is acceptable. With the post-training INT8 quantization provided by ONNX Runtime, the resulting improvement was significant: both memory footprint and inference time were brought down to about a quarter of the pre-quantized values, comparing to the original model with an acceptable 3% reduction of model accuracy. You can find details of the model architecture design, hyperparameter tuning, accuracy and performance in the research paper<\/a> we published at the 2019 KDD conference.<\/p>\n

The final gate of the release to production was doing online A\/B experimentation comparing the new LSTM model and the previous production model. The online A\/B experimentation results in the Table 2 below showed about 25% improvement on top-1 recommendation precision (the precision of the first recommended completion item in the completion list) and 17% improvement on mean reciprocal rank<\/a>(MRR), which convinced us the new LSTM model is significantly better than the previous production model.<\/p>\n

\"A\/B
Table 2. A\/B testing result showing significant metrics’ improvements<\/figcaption><\/figure><\/p>\n

 <\/p>\n

Python developers: Try IntelliCode completions<\/strong> and send us your feedback! <\/strong><\/span><\/p>\n

With a great team effort, we completed the staged roll-out of the first deep learning model to all the IntelliCode Python users in Visual Studio Code. In the latest release of the\u00a0IntelliCode extension for Visual Studio Code<\/a>, we\u2019ve also integrated ONNX Runtime and the LSTM model to work with the new Pylance<\/a> extension, which is written entirely in TypeScript. If you\u2019re a Python developer, please install the IntelliCode extension and provide us feedback.<\/p>\n

What\u2019s next?<\/strong><\/span><\/p>\n

We are looking forward to shipping the deep learning model of member completion for more programming languages in IntelliCode\u2019s coming releases. \u00a0In the meantime, we are actively working on more advanced transformer based deep learning model<\/a> for even longer code completions.<\/p>\n

How can you leverage what we\u2019ve learned?<\/strong><\/span><\/p>\n

Along this journey, ONNX Runtime and Azure Machine Learning were critical in making these developments possible. You can learn how to leverage them in your own scenarios from these links (ONNX Runtime<\/a>, Azure Machine Learning<\/a>). The ONNX and Azure Machine Learning teams will be happy to hear your feedback!<\/p>\n","protected":false},"excerpt":{"rendered":"

After\u00a0leveraging technologies like\u00a0Azure Machine Learning\u00a0and\u00a0ONNX Runtime,\u00a0IntelliCode has\u00a0successfully\u00a0shipped\u00a0the first\u00a0deep learning model for all the\u00a0IntelliCode\u00a0Python\u00a0users in Visual Studio\u00a0Code.\u00a0This blogpost gives a detailed account of the journey from research to model deployment.<\/p>\n","protected":false},"author":33576,"featured_media":230358,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[155],"tags":[6790,6789,6784,6786,4469,467,6785,1054,6787,6788,526],"class_list":["post-230355","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-visual-studio","tag-azure-machine-learning","tag-azure-ml","tag-data-science","tag-deep-learning","tag-developer","tag-intellicode","tag-intellicode-completions","tag-ml","tag-onnx","tag-onnx-runtime","tag-productivity"],"acf":[],"blog_post_summary":"

After\u00a0leveraging technologies like\u00a0Azure Machine Learning\u00a0and\u00a0ONNX Runtime,\u00a0IntelliCode has\u00a0successfully\u00a0shipped\u00a0the first\u00a0deep learning model for all the\u00a0IntelliCode\u00a0Python\u00a0users in Visual Studio\u00a0Code.\u00a0This blogpost gives a detailed account of the journey from research to model deployment.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/posts\/230355","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/users\/33576"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/comments?post=230355"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/posts\/230355\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/media\/230358"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/media?parent=230355"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/categories?post=230355"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/tags?post=230355"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}