Intelligent Visual Studio Search Service
In Visual Studio (VS) 2019 v16.0 release, the Visual Studio Search shortcut Ctrl+Q, as shown in the screenshot below, was introduced as a Quick Launch for VS feature searches and expanded in v16.3 to support code searches. With millions of developers using the VS IDE, more and more developers are using this hotkey to boost their productivity; we hear customer feedback at developer conferences, customer interviews, and user surveys regarding improving search relevancy and increasing feature discoverability.
Looking into the existing in-box Ctrl+Q search experience, there are a few limitations:
- The search results relevancy was built-in to VS releases and hard to improve over time.
- Relies on keywords manually added to each feature for keyword matching.
- Employed heuristics that were based on point-in-time data for ranking algorithm tuning.
To address these limitations, last year we started to explore a solution built upon AI/ML based Azure services and shipped the feature in v16.9 release – an intelligent Visual Studio feature search service powered by Azure services, currently focusing on VS feature searches, and integrated with the Ctrl+Q box. In this blog, we want to share our experience and learnings in building the feature search service.
The system extracts Visual Studio metadata for all 14 locales of each release (e.g., v16.9 release), uploads the metadata to a secure store for further validating and processing; the processed data is fed into the index processing pipeline, which leverages MS language analyzers for different locales and MS text analytics to extract keywords for VS features. When a user performs a search in the Ctrl+Q box in the VS IDE, the request is sent through the load balancer to one of the nearest geo-redundant REST API endpoints, which processes the request based on user’s search context, and returns the top matched search results to the user.
There are two primary data flows in this system, each consisting of various components and Azure services. Currently the system supports all 14 Visual Studio locales and v16.9+ releases.
From top left: feature metadata generation and index creation
- An automated pipeline extracts Visual Studio features metadata for all locales of each release and uploads them into a secured endpoint.
- An Azure Functions app triggers, validates, and transforms the latest drop of the new or updated VS feature metadata, as well as a Synonyms mapping list, then stores the processed data into Azure Storage service’s Table storage.
- The indexers of the Azure Search service pick up the updates in storage to index all documents, while leveraging Azure Cognitive Services’ NLP text analytics for keywords extractions with language detection during indexing process and query result retrieval. For any updated metadata, we do incremental caching with the Azure Storage to only process those features with changes.
From the right: feature search and data retrieval
- When a Visual Studio user types in a search query in Ctrl+Q box, the VS client sends the anonymized request to the feature search service, going through the Azure Front Door that routes the call into the nearest region for the REST API endpoints.
- The REST API app receives the search request, checks to see if the search results were in local cache or in the Redis Cache, and quickly returns it. If not yet cached, it retrieves new results from one of the indexes in the Azure Search service, which in turn will be cached for future fast retrieval.
- Based on the user’s search context, such as whether any project was opened or while Visual Studio is in debugging mode, the top results are then compared to the contextual ranking data with same feature identifiers in the Re-ranking model. The contextual ranking data was pre-processed and cached to speed up the process.
- The top re-ranked results then return to the Visual Studio user.
- The anonymized user’s feature search click-through and contextual usage data are fed into the Contextual Usage store for continued contextual ranking updates in the re-ranking model.
Search Scoring and Ranking
To reach the parity of the existing in-box search experience and improve upon the search relevancy, we took a few calculated choices in the index design, ranking algorithm, and scoring tuning:
- Created a few composable cognitive skills to extract the key phrases from each VS feature with detected language for where feature keywords were left off.
- Added the Synonyms map based on historical search usage, which is supported in the English locale index in this release.
- Designed the scoring profile with selected fields and tag boosting based on whether the features were menus, commands, options, project and item templates, preview features, or setup items.
- Adapted MS Language Analyzers in our indexes for lemmatization and word decompounding that are more suitable for different VS locales.
- Leveraged Azure Search service’s feature to compute ranking scores of documents across all 12 shards in the Azure Search service for consistent search results of those features that are closely related but scattered in different shard.
- Used the BM25 Similarity ranking function with term frequency hyperparameters customization, instead of the default TF-IDF similarity. The latter is not desirable for us because certain categories of VS features that are infrequently used, such as Installer components, are biased towards due to their greater lengths.
- Built a Lucene Query function to support fuzzy, proximity, wildcard searches for greater search flexibility.
Service in Production
To achieve production level performance, scalability, geo-redundancy, and reliability, manage potential service throttling, and monitor service health, the system is plugged into our well-tested AI service infrastructure, which supports our service’s development, staging and deployments with Azure DevOps and Azure Resource Manager templates with consistency.
Results and Benefits
So how did the search relevancy from the search service perform compared to that of the inbox search experience? We conducted a few staged A/B experiments in v16.8 release and measured with MRR the scores of the Control group and the Treatment group, and statistically significant improvements in the Treatment group were consistently observed at different stages.
Below are a few examples of the top searched VS features, their relevant results without and with the search service.
When searching for “solution”, the topmost selected result is “Solution Explorer”. The left screenshot below without the service has the result at #5 in the result list, while it is at the top 1 with the service shown in the right screenshot below.
When searching for “keyboard”, we got frequent user feedback about wanting “keyboard” mapped to “Change keyboard settings”, with the service, the feature result is showing at the top 1 in the right screenshot.
With the release of the VS feature search service powered by Azure services, the system enables:
- Preliminary context aware search experience.
- Iterative, data-driven improvements to feature search rankings.
And we can help you with more and more accurately what you are looking for with faster iterations in improvements.
There are several ways that we can continue to improve the system. For example, the system can consider the current coding file’s language, the current project’s type, what workloads are currently installed in the VS IDE, to setup automatic feedback loop based on user behavior, and provide more focused search results.
Taking search a little further, the system can be scaled to provide results from additional data sources, such as VS online docs and tech blogs, Bing search, etc., for improved VS feature discoverability, to unify with AI-powered code search services, and to eventually offer personalized search experience in VS.
Help Us Improve with Your Feedback
User feedback is the best way for us to improve the results. We encourage you to install the latest v16.9 release to try Ctrl+Q with our search service, as shown in the screenshots in the Results section above, the little cloud icon at the end of each result indicates that the result was from the search service.
We invite you to share your feedback or ideas for improvements at Suggest a Feature for Visual Studio. On behalf of our team working on enabling this feature, thank you!