Status on Visual Studio feature suggestions

Mads Kristensen

Mads

Visual Studio receives over 500 feature suggestions from customers every month on the Developer Community website. Handling that amount is a huge effort and we’d like to share with you how we handle this volume and the steps that we take to respond to them all. What happens to suggestion tickets after they’re opened, how many make it into Visual Studio, and what happens to the rest? Let’s find out.

Let’s start with the breakdown of incoming suggestion tickets in the past 6 months and what state they are in today. We find that around 15% of the suggestions are challenging to act on, and they typically fall into the following buckets.

11% – Closed as duplicate
3% – Closed due to missing info from customer
1% – Closed because they were not suggestions for Visual Studio

We do our best to follow up with customers to get more information where we can and move them into the next stage. For example, when making a suggestion to add a command to a context menu, it is important for us to know which context menu you meant.

That leave us with 85% left which are currently moving their way through the system. Here is the status of those tickets currently in our system:

40% – Closed for a number of reasons (more info below)
20% – New, not yet processed or triaged
28% – Under review and gathering votes and comments
3% – Awaiting more info from customer
3% – On roadmap (under development)
6% – Completed and released

Now let’s dig in and see what’s behind those numbers.

From New to Under Review

We have a filtering system that automatically routes incoming suggestions to the appropriate team within the Visual Studio organization. Within my team, we have established a weekly process to triage these routed suggestions and review status. The process we follow looks like this:

  1. Does this bug belong to my team?
    • If not, move it to the right team
  2. Is the suggestion a duplicate of an existing suggestion?
    • If so, close it and transfers all votes to the original ticket (happens automatically)
  3. Does the suggestion contain all needed information?
    • If not, ask customer for more information
  4. Was this suggestion already completed or in active development?
    • If so, close it as either Completed or On Roadmap
  5. If it made it this far, mark it Under Review to gather votes and comments for 90 days

By following these steps, most suggestions end up Under Review as we gather more data, refine any repos or requirements. These make up over a quarter of all suggestions.

Every time someone adds a new comment to an existing ticket, we receive an email, so we know what’s going on with each ticket along the way, and can respond if needed.

Moving on from Under Review

Within 90 days, we attempt to address items that are still marked Under Review. Our options are:

  1. Mark it as Completed because we implemented the suggestion
  2. Mark it as On Roadmap because it’s in active development or will be very soon
  3. Close it because it didn’t get any votes and/or we’re not able to prioritize it

When we implement a suggestion, we mark it Completed or On Roadmap. Currently, approximately ~10% of the incoming suggestions go on to be implemented or added to the roadmap.

But what about the ones that don’t?

Reasons for closing suggestions

Most suggestions are good suggestions and it’s always painful to close them. Especially because a lot of them are some that we personally would like to see implemented. As developers, you know that time and resources are finite, which means we can’t implement all suggestions.

The reason we close suggestions is a mix of multiple factors, such as:

  1. It didn’t receive any votes after 90 days as Under Review
  2. It got a few votes, but an implementation will not fit within our available resources
  3. It involves areas in Visual Studio that see little usage by our customers
  4. It has negative side-effects such as degraded performance, accessibility etc.

Over a third of all suggestions end up closed due to one or more of the above reasons.

On the positive side, even for some suggestions that we close, we do move the capability into an experimental extension for Visual Studio. This allows us to lower the cost of delivering a quality product investment, and where we can draw more interest from the community.

Suggestion completed

6% of all actionable suggestion tickets end up marked as Completed. It may not sound like much, but it is about 1 suggestion per weekday. Let that sink in. Every single weekday, the Visual Studio engineering team implements a community submitted suggestion.

Before we implement a suggestion, we first write a spec for it if needed. Then we schedule the work item in a sprint for engineering to pick up. The implementation sometimes require work by multiple teams to coordinate, so they can each do their piece of the feature.

After automated test and compliance runs have finished, it’s time for code review before the code starts its journey toward the Visual Studio master branch. More automated testing runs and finally manually testing follows. After fixing all identified bugs, the completed suggestion makes its way to a Visual Studio Preview release for user testing and stabilization.

So, how do we decide to implement suggestions and how can you optimize the chances of your suggestion making it? We look at several things:

  1. Suggestions with many votes and continuous votes over time
  2. Suggestions in areas that see lots of usage by our customers
  3. Suggestions that are easier to implement
  4. Suggestions that would improve Visual Studio’s competitive advantage
  5. Well written suggestion with all relevant information in the description

A different way to think about it is to turn it around. Imagine someone wanted you to implement a feature in your product. It’s in the best interest of our product and customers to complete as many suggestions as possible, and we strive to do so.

The best times are when we get to make a lot of people happy with a feature implementation based on a suggestion.

We can must do better

We’ve gotten feedback that this process feels like a black box. Customers feel like they don’t get a response and they don’t know the status of their suggestions.

After submitting a suggestion, there is no transparency into the process, and it ends up closed without any good reason 6 months later. I end up feeling frustrated and angry. I don’t want to submit another suggestion just to be ignored. – Anonymous Visual Studio user

This is not acceptable. We must do better.

Some ideas that we are working on within the team are as follows. And, we welcome your feedback on what we might do more of to help you understand the process better.

First up, we want to be much more transparent about the process. That’s exactly what this blog post aims to achieve.

Secondly, we must be faster at responding to new suggestions. That means triaging them within the first week, so we can bring down the 20% of new untriaged suggestions to a minimum. It also means not leaving any suggestions to linger for months. This will add visibility into what is going on with the suggestions much earlier and throughout its various phases. We’ve made great progress with this in the past 6 months, but still have a bunch of open tickets to go.

Thirdly, we need to be better at giving reasons for closing tickets. Individually written by the program manager that closed them and not an automated response. As we’re getting better at handling the vast amount of incoming suggestions, this is where we’ll focus next.

Feedback

I hope this blog post helps shed light on the way we handle suggestion and how we plan to improve. Completing a suggestion every single weekday will hopefully encourage you to continue opening suggestion tickets.

In closing, we’d really like to hear your thoughts or questions. What could we do better and what do we do well? Let us know in the comments below.

Mads Kristensen
Mads Kristensen

Senior Program Manager, Visual Studio Extensibility

Follow Mads   

102 comments

Comments are closed.

  • Avatar
    Jon

    As many others have covered, the feedback and bug-reporting UI/UX is terrible, using the lack of votes to close good suggestions and valid bugs is ridiculous (who has time to wander that terrible site tossing around votes?), but the reason I’ve stopped using the site is when a valid bug is closed either because nobody got around to working on it, or worse, somebody arbitrarily decides it isn’t a bug. A prime example is this bug I reported where simply running VS continuously generates a ton of errors in the event log. Somebody (probably an intern or contractor judging from the inept answer) pops in to post irrelevant junk he randomly Googled (he literally writes “I looked up these error codes on the internet”) then closes it, probably one ticket closer to his daily quota. Guess what? It still happens in VS2019. This was the straw that broke the camel’s back for me. I haven’t reached the point of giving up on VS yet, but I’m done helping you debug your product.

  • Avatar
    Mike Diack

    I’ve said it elsewhere, but having looked at how this thread is expanding I went and looked at some of the bugs people raise. A few trends are very clear.1) Versions that a fix is in are rarely clear – the answer is often “fixed in latest release”. For the love of God, please put the version number in – surely it can’t be difficult for support staff (or the bot that does it), to put the version number of the fix in?2) Too often, the replies are just verbatim that could be written by a robot, and are more like a press release/lawyer – “your feedback is important” etc. and convey no useful information.
    3) As others have mentioned there have been important bugs floating around for 2-3 years, in a weird state of limbo (open, but not closed, fixes working for some people not others, lots of “me too” or “still broken in version ……” . Surely you can do better than this. Its just a mess. But my main bugbear is 1 and 2.

  • Avatar
    Mike Diack

    One practical piece of advice for others posting here – email John Montgomery and/or Nick Uhlenhuth with your issues – once I started bugging them directly via email with examples of the mess that VS 2017 and the bug reporting/follow up/tech support process is, things started to happen.

  • John B
    John B

    The biggest frustration is that first responses are from people who jump straight to “need more info” even for simple reports. It gives the impression they’re not versed in the technology, aren’t taking time to read, or lack communication skills.

    • Avatar
      Chuck Ryan

      Yeah, they love to give you a research project to figure out what the problem is in their product and then act suprised when you ask where to send your billable hours.
      Do they think we work for free?

  • Avatar
    Neil MacMullen

    Well, I’ve just had yet another issue-report closed with the standard “lack of activity” template reply.  I really do appreciate the necessity to prioritize and the reality of finite development resource but I’ve now reported several genuine bugs (at the the very least, undesirable behaviours) that have been ignored this way.  In the development teams I’ve run, we always treated customer reports as an important feedback mechanism to uncover bugs that had escaped internal testing and actively chased these up rather than just waiting to see if more than one customer would notice.  (On more than a few occasions a ‘minor’ behavioural oddity turned   out to be an indicator or something potentially far more serious.)  I don’t know whether the issue is just that the people doing the triaging are primed to try and ‘protect’ core development but I’d echo a few of the other comments here that the impression I generally get is that the triage process seems to be heavily biased towards rejection – both of false reports and genuine ones.  Often there is a request to provide infeasible amounts of supporting evidence (yes – we all want a local repro case – I understand!).  I can’t just hand over my entire solution (or PC!) but in the past I’ve offered to run test builds to capture problems and never been taken up on it.  Anway, I really do hope things improve.  Visual Studio is still actually a great product and I have very few gripes with it  but the feedback process is an exercise in frustration and I’m inclined to stop providing feedback since the vast majority is rejected.

  • Avatar
    Mike Diack

    Continuing my comments about the terrible feedback loop on bug reports (specifically identifying which “latest release” has the fix, by absolute number, rather than just saying “Fixed in latest release” (at the moment that the Microsoft staff member wrote the bug report update!)Perfect example:The following bug was fixed in this week’s Visual Studio Update 9.15 patch:

    https://developercommunity.visualstudio.com/content/problem/608423/calling-pmr-monotonic-buffer-resource-release-will-1.html
    Release notes here:
    https://docs.microsoft.com/en-us/visualstudio/releasenotes/vs2017-relnotes#-visual-studio-2017-version-15915-BUT:Where on that bug report, does it make ANY mention of a fix being in the 15.9.15 update?
    Nowhere!

  • Avatar
    SuperCocoLoco .

    Microsoft is clearly ignoring his user base feeback, suggestions and bug reports. Reporting suggestions or bug are a waste of time.
    The best thing to describe this is all suggestions post about the “New Project” window dialog. There are pages with posts about this in the developer community claiming for restoring the old tree-based “New Project” window dialog and also ensuring that it is the worst thing Microsoft has done in Visual Studio since 1997. Also some posts have a lot of votes and comments.
    But the only answers from Microsoft are they are “improving” the “New Project” window dialog adding more filters, more search options that absolutely nobody has claimed. Users are only requesting and reclaiming the older TREE-BASED new project dialog. But Microsoft clearly said that do not restore the old tree-based window dialog never again, ignoring, angering and infuriating his entire user base.
    But not only not restoring the old tree-based new projhect window dialog, they are also removing in 16.3 the underlaying code that allow thrid party extensions to use that good features that users reclaims, like the old “New Project Dialog” and the old “Start Page” for more angering and infuriating your entire user base.

    Why is Microsoft ignoring feedback and user base? What about transparency? Why Microsoft not admit how bad do it with the “New Project” dialog, the new “Start Window” and the new “Compact Menu”?
    More things of posting in developer communty are a lot of post that Microsoft changes the title, like my title`aprox. “Allow to show the complete Title Bar option” in Visual Studio 2019.0 preview changed to “Fix usability issues in compact menu bar” that is not the same thing and has nothing to do with my suggestion, because i have no interest in compact menu and don’t want Microsoft fixes any issues, i only want the real and full Title Bar like any other application. In the last minute Microsoft added that option in the final version.
    More things are closing posts and suggestions as duplicate, linking to other suggestions that has nothing to do and are clearly not the same.
    More things are a clearly and reproducible bug in Visual Studio 2017.0 WPF editor, that changing the name of a control produce a lot of errors because code not find the old name of the control and closed because didn’t receive any votes. Finally fixed more than a year later in 2017.9 aprox.
    The way Microsoft handle suggestion is an horrible mess, not really hearing his user base feedback and clearly a waste of time. I will never report any suggestions or errors ever again.

  • Avatar
    moovthis

    I don’t think this is necessarily a problem with the feedback system. My thinking is that there are underlying causes.
    First the culture at Microsoft is to come up with new and more efficient means to do stuff. Understood. The end users want a stable, bug free and fast development platform (and the tooling that goes with it) that is around for a long time AND has a low cost of ownership. For my purposes the Community Edition works just fine so the initial cost of the product is fine (i.e. free), but I measure my time wasted on what I describe here in hours and sometimes days. And so far I have stuck with a single development platform and as such have avoided months, maybe a year of my time redoing my code because of the new flavor of the year platform from Microsoft. 
    Second, a lack of understanding how Microsofts customer’s use case is different. Microsoft is working on a product but the end users buy a tool. I think that means that the knowledge about the product at Microsoft has to be at a higer level than the users of the tool. As result minor changes made by Microsoft often times result in problems for the users that take a significant effort to resolve. I have several examples but this one is very typical. Some functionality got added, causing thousands of exceptions that prevented me from building and deploying my code and several hours of my time to understand what this was about and how to fix it. More recently, Microsoft thought it was prudent to remove a registry key in my VS2017 install when the 16.2.3 update to VS2019 gets installed. That totally effed up EF in VS2017 and again several hours of my time down the drain. Someone should have considered the consequence of turning on this feature. In addition consider those updates that are forced down our throats at the most inopportune moments where the functionality of the machine at the very least is severly degraded at worst unusuable.
    Third, there has to be someone to put a stop to hare-brained ideas, to nip things in the bud, an adult in the room. Let’s take the debugger step icons in VS2019 as an example. Someone took something that worked and probably no one was complaining about and changed it into something that was regressive in nature and resulted in complaints and jokes (in general about the Microsoft icon do-over). I am thinking this is an example where the adult in the room should have acted and the responsible persons punished (no use of smileys and emojis for a month?).
    From my perspective I cannot rationalize the things I see happening with Microsoft except to think Microsoft does not use their own software (like I do) and that there is a large quantity of kid coder employees who have no clue of the consequences of their actions, and a shortage of adults in the room.
    I cope with this dichotmoy by laughing out when the latest Microsoft blog post comes around and explains code efficiency improvements measured in seconds, new icons, glowing stories about emojis, smileys, transparency and like buttons.  Of course sharing the pain helps with coping so I have bi-weekly sessions with developer friends where we exchange the lastest frustrations with Microsoft. 
    I wish for Microsoft to consider the total cost of ownership of their products and act cautiously as to not waste our time. The issues with the feedback system are a result of some of the issues I have addressed. 
    This is just my opinion, I could be wrong.

  • Avatar
    Michael DeMond

    Another dubious example of “improvement.”  You ask users to provide information with a feedback tool, which takes many minutes to complete.  And THEN when the issue is “triaged” you further ask the user to jump through MORE hoops to collect more information, that for some reason was not collected in the first place?!  Worth noting that JetBrains was able to fix an issue with the provided collected information whereas MSFT cannot.  *shrug* https://developercommunity.visualstudio.com/content/problem/697688/closed-visual-studio-and-now-its-frozen-on-unloadi.html