Visual Studio receives over 500 feature suggestions from customers every month on the Developer Community website. Handling that amount is a huge effort and we’d like to share with you how we handle this volume and the steps that we take to respond to them all. What happens to suggestion tickets after they’re opened, how many make it into Visual Studio, and what happens to the rest? Let’s find out.
Let’s start with the breakdown of incoming suggestion tickets in the past 6 months and what state they are in today. We find that around 15% of the suggestions are challenging to act on, and they typically fall into the following buckets.
11% – Closed as duplicate 3% – Closed due to missing info from customer 1% – Closed because they were not suggestions for Visual Studio
We do our best to follow up with customers to get more information where we can and move them into the next stage. For example, when making a suggestion to add a command to a context menu, it is important for us to know which context menu you meant.
That leave us with 85% left which are currently moving their way through the system. Here is the status of those tickets currently in our system:
40% – Closed for a number of reasons (more info below) 20% – New, not yet processed or triaged 28% – Under review and gathering votes and comments 3% – Awaiting more info from customer 3% – On roadmap (under development) 6% – Completed and released
Now let’s dig in and see what’s behind those numbers.
From New to Under Review
We have a filtering system that automatically routes incoming suggestions to the appropriate team within the Visual Studio organization. Within my team, we have established a weekly process to triage these routed suggestions and review status. The process we follow looks like this:
- Does this bug belong to my team?
- If not, move it to the right team
- Is the suggestion a duplicate of an existing suggestion?
- If so, close it and transfers all votes to the original ticket (happens automatically)
- Does the suggestion contain all needed information?
- If not, ask customer for more information
- Was this suggestion already completed or in active development?
- If so, close it as either Completed or On Roadmap
- If it made it this far, mark it Under Review to gather votes and comments for 90 days
By following these steps, most suggestions end up Under Review as we gather more data, refine any repos or requirements. These make up over a quarter of all suggestions.
Every time someone adds a new comment to an existing ticket, we receive an email, so we know what’s going on with each ticket along the way, and can respond if needed.
Moving on from Under Review
Within 90 days, we attempt to address items that are still marked Under Review. Our options are:
- Mark it as Completed because we implemented the suggestion
- Mark it as On Roadmap because it’s in active development or will be very soon
- Close it because it didn’t get any votes and/or we’re not able to prioritize it
When we implement a suggestion, we mark it Completed or On Roadmap. Currently, approximately ~10% of the incoming suggestions go on to be implemented or added to the roadmap.
But what about the ones that don’t?
Reasons for closing suggestions
Most suggestions are good suggestions and it’s always painful to close them. Especially because a lot of them are some that we personally would like to see implemented. As developers, you know that time and resources are finite, which means we can’t implement all suggestions.
The reason we close suggestions is a mix of multiple factors, such as:
- It didn’t receive any votes after 90 days as Under Review
- It got a few votes, but an implementation will not fit within our available resources
- It involves areas in Visual Studio that see little usage by our customers
- It has negative side-effects such as degraded performance, accessibility etc.
Over a third of all suggestions end up closed due to one or more of the above reasons.
On the positive side, even for some suggestions that we close, we do move the capability into an experimental extension for Visual Studio. This allows us to lower the cost of delivering a quality product investment, and where we can draw more interest from the community.
Suggestion completed
6% of all actionable suggestion tickets end up marked as Completed. It may not sound like much, but it is about 1 suggestion per weekday. Let that sink in. Every single weekday, the Visual Studio engineering team implements a community submitted suggestion.
Before we implement a suggestion, we first write a spec for it if needed. Then we schedule the work item in a sprint for engineering to pick up. The implementation sometimes require work by multiple teams to coordinate, so they can each do their piece of the feature.
After automated test and compliance runs have finished, it’s time for code review before the code starts its journey toward the Visual Studio master branch. More automated testing runs and finally manually testing follows. After fixing all identified bugs, the completed suggestion makes its way to a Visual Studio Preview release for user testing and stabilization.
So, how do we decide to implement suggestions and how can you optimize the chances of your suggestion making it? We look at several things:
- Suggestions with many votes and continuous votes over time
- Suggestions in areas that see lots of usage by our customers
- Suggestions that are easier to implement
- Suggestions that would improve Visual Studio’s competitive advantage
- Well written suggestion with all relevant information in the description
A different way to think about it is to turn it around. Imagine someone wanted you to implement a feature in your product. It’s in the best interest of our product and customers to complete as many suggestions as possible, and we strive to do so.
The best times are when we get to make a lot of people happy with a feature implementation based on a suggestion.
We can must do better
We’ve gotten feedback that this process feels like a black box. Customers feel like they don’t get a response and they don’t know the status of their suggestions.
After submitting a suggestion, there is no transparency into the process, and it ends up closed without any good reason 6 months later. I end up feeling frustrated and angry. I don’t want to submit another suggestion just to be ignored. – Anonymous Visual Studio user
This is not acceptable. We must do better.
Some ideas that we are working on within the team are as follows. And, we welcome your feedback on what we might do more of to help you understand the process better.
First up, we want to be much more transparent about the process. That’s exactly what this blog post aims to achieve.
Secondly, we must be faster at responding to new suggestions. That means triaging them within the first week, so we can bring down the 20% of new untriaged suggestions to a minimum. It also means not leaving any suggestions to linger for months. This will add visibility into what is going on with the suggestions much earlier and throughout its various phases. We’ve made great progress with this in the past 6 months, but still have a bunch of open tickets to go.
Thirdly, we need to be better at giving reasons for closing tickets. Individually written by the program manager that closed them and not an automated response. As we’re getting better at handling the vast amount of incoming suggestions, this is where we’ll focus next.
Feedback
I hope this blog post helps shed light on the way we handle suggestion and how we plan to improve. Completing a suggestion every single weekday will hopefully encourage you to continue opening suggestion tickets.
In closing, we’d really like to hear your thoughts or questions. What could we do better and what do we do well? Let us know in the comments below.
Another dubious example of "improvement." You ask users to provide information with a feedback tool, which takes many minutes to complete. And THEN when the issue is "triaged" you further ask the user to jump through MORE hoops to collect more information, that for some reason was not collected in the first place?! Worth noting that JetBrains was able to fix an issue with the provided collected information whereas MSFT cannot. *shrug* https://developercommunity.visualstudio.com/content/problem/697688/closed-visual-studio-and-now-its-frozen-on-unloadi.html
I don't think this is necessarily a problem with the feedback system. My thinking is that there are underlying causes.
First the culture at Microsoft is to come up with new and more efficient means to do stuff. Understood. The end users want a stable, bug free and fast development platform (and the tooling that goes with it) that is around for a long time AND has a low cost of ownership. For my purposes...
Microsoft is clearly ignoring his user base feeback, suggestions and bug reports. Reporting suggestions or bug are a waste of time.
The best thing to describe this is all suggestions post about the "New Project" window dialog. There are pages with posts about this in the developer community claiming for restoring the old tree-based "New Project" window dialog and also ensuring that it is the worst thing Microsoft has done in Visual Studio since 1997....
Continuing my comments about the terrible feedback loop on bug reports (specifically identifying which "latest release" has the fix, by absolute number, rather than just saying "Fixed in latest release" (at the moment that the Microsoft staff member wrote the bug report update!)Perfect example:The following bug was fixed in this week's Visual Studio Update 9.15 patch:
https://developercommunity.visualstudio.com/content/problem/608423/calling-pmr-monotonic-buffer-resource-release-will-1.html
Release notes here:
https://docs.microsoft.com/en-us/visualstudio/releasenotes/vs2017-relnotes#-visual-studio-2017-version-15915-BUT:Where on that bug report, does it make ANY mention of a fix being in the...
Well, I've just had yet another issue-report closed with the standard "lack of activity" template reply. I really do appreciate the necessity to prioritize and the reality of finite development resource but I've now reported several genuine bugs (at the the very least, undesirable behaviours) that have been ignored this way. In the development teams I've run, we always treated customer reports as an important feedback mechanism to uncover bugs that had escaped internal testing...
Yeah, same here. The only response I received to my report was from Jane Wu saying they were shutting it down as low priority since there was no redeeming activity or whatever… that was my last attempt at a bug report as I have no interest in wasting my time. Although I must admit some suggestions have great comedic value. I call this one Pratik’s one man show: https://developercommunity.visualstudio.com/idea/556836/visually-indicate-new-templates-after-installing-a.html
ha-ha -that’s great! I guess there are some perks to working on the dev team 😉
The biggest frustration is that first responses are from people who jump straight to “need more info” even for simple reports. It gives the impression they’re not versed in the technology, aren’t taking time to read, or lack communication skills.
Yeah, they love to give you a research project to figure out what the problem is in their product and then act suprised when you ask where to send your billable hours.
Do they think we work for free?
One practical piece of advice for others posting here – email John Montgomery and/or Nick Uhlenhuth with your issues – once I started bugging them directly via email with examples of the mess that VS 2017 and the bug reporting/follow up/tech support process is, things started to happen.
I've said it elsewhere, but having looked at how this thread is expanding I went and looked at some of the bugs people raise. A few trends are very clear.1) Versions that a fix is in are rarely clear - the answer is often "fixed in latest release". For the love of God, please put the version number in - surely it can't be difficult for support staff (or the bot that does it), to...
All valid points
As many others have covered, the feedback and bug-reporting UI/UX is terrible, using the lack of votes to close good suggestions and valid bugs is ridiculous (who has time to wander that terrible site tossing around votes?), but the reason I've stopped using the site is when a valid bug is closed either because nobody got around to working on it, or worse, somebody arbitrarily decides it isn't a bug. A prime example is this...
You all put something “Under Consideration” and then mark it for closure 60 days later? whut?
https://developercommunity.visualstudio.com/content/problem/600282/solution-explorer-dependencies-shows-warning-icon.html
Also, do you all actually enjoy making counter-intuitive user interfaces now or are you there just to collect a paycheck these days and chalk it up to something Google would do? All of the above?