How we use User Voice to make a better product

To increase our engagement with our customers, we created the Visual Studio User Voice site to capture your feedback for Team Foundation Server (TFS) and Visual Studio Online (VSO). Since then more than 20,000 people have been participating and we have received over 2,300 suggestions. In the last year we have completed 150 suggestions. We are excited to have so many people who have taken the time to give us feedback. The goal of this post is to describe how we use the User Voice data, and what we have changed to improve our response to the User Voice feedback. If you think we can improve how we use User Voice, please leave a comment or reach out to me via email: ewald.hofman at microsoft.com.

Gaining insight

When we started with User Voice, we received a handful of suggestions per week. Many of these items were already on our backlogs and to be as transparent as possible to our supporters we marked these suggestions as “Under Review”. However we haven’t made any progress on them yet and as a result we haven’t responded to these suggestions for more than 4 years. It may not be a surprise that we created the perception of not listening to customers. We felt really bad about it, so late 2013 we started an initiative to improve how we interact with User Voice and its community.

The first step in the process was to groom the list of User Voice suggestions. We merged duplicates, updated the status of already completed suggestions and assigned the most popular suggestions to the correct feature teams. When we shared the top 20 most popular suggestions, trends and whether we were addressing them for the next major release with all the feature teams, it generated a lot of interest about User Voice. And we didn’t stop with that one time exercise, we continued to groom the User Voice suggestions to keep the list healthy, and regularly we share a summary report to all feature teams working on TFS/VSO.

Giving insight

After we gained the insight in the User Voice data, it was time for the next step. Working with User Voice was a one-way communication: we listened to the community, but did not update the status of completed suggestions nor did we tell you if we were working on something. To improve in this area we made two changes.

1. Track suggestions on our backlogs: We did not complete the User Voice suggestions, because we did not have a link between our backlog and User Voice. Therefore the feature team was just not aware that a shipped feature was backed by User Voice. To improve that we created bi-directional links between our backlogs and the User Voice for the most popular suggestions. When we ship a feature we can easily find the User Voice suggestions that we need to update. If you see an open suggestion which you think is already completed, please let me know and I will get it fixed.
2. Frequent updates on in-progress items: The “Under Review” state was overloaded because we used it for so many suggestions in the past. To better indicate which one we are actively working on or which ones are on our 6-month plan, we introduced the states “Started” and “Planned”. The goal is to give an update on in progress suggestions at least every 3 months. The majority of the Under Review suggestions are still on our backlog, but they are not part of the 6-month plan. We are going to bite the bullet and reset the state of all these Under Review suggestions, to make it very simple to you. If a suggestion has a status, you can expect us to give regular updates until we have completed the item.

We are listening…

To prioritize our backlogs we use many channels for input, and User Voice is one of these channels. And we take it very seriously.

• Below you see the list of 150 suggestions that we have completed in the last year. Based the User Voice feedback we completed items which were not in our original plans, including the “Creating Queries – Token for Current Iteration” and “ability to add multiple activities per team member in sprint”. But we heard you loud and clear and we prioritized these items higher on our backlogs.
• There are currently 49 items in progress (state is “Planned” or “Started”). Some of them have gone stale, but in the next few weeks all in progress suggestion will have an recent update. You can hold us accountable for that, so if you see an in-progress suggestion without a recent update don’t hesitate to reach out to me.
• Recently Brian Harry has published a blog post which shows some of the items that we are planning to work on in the next few months, and that list is influenced by User Voice.
• In each planning cycle the feature teams are being questioned why they are not working on one of the top 10 items

These are the suggestions that we have closed in the last year sorted on the number of votes.

Suggestion	Votes
rename project in TFS	6159
Add “Create Unit Tests” menu option back!	3038
Creating Queries – Token for Current Iteration	2188
Make Code Review available to Professional users	1221
Customization of TFS Web Access	1201
ability to add multiple activities per team member in sprint	1094
Allow TFS Task Board to have custom colors	880
In the team Explorer, Change Default State value of a selected Work Item to “Associate” instead of “Resolve” on Check In	845
Excel integration with MTM	792
Allow tests to be grouped by class, category, and namespace in Test Explorer	699
Support more browsers and browser versions in MTM	520
Integrate Team Wiki / project documentation pages	471
Team Test – Improve the usability of Microsoft Test Manager	457
Bring back the view all pending changes window (VS11)	436
Use a build controller over more than one Team Project Collections	397
Provide the ability to prioritise Backlog items in a sprint backlog by dragging and dropping the item	340
Provide TF build activities like “deploy web app”, “deploy database”, or “copy file”	305
Allow tests to be paused, allow other tests to be run, and then resume the paused test case	293
Suite created by “Workitem and direct links” query in MTM	282
Implement TFS Build for linux (Create Linux build agent)	272
Continue support for Database Unit Tests in VS 2012	267
Testers should be able to get parameters for a test case from a csv file	260
Show me all test suites/ test plans a test case belongs to	245
Provide versioning of test cases to support multiple parallel releases	243
Allow testers to edit test cases inside Manual Test Runner	241
Support Windows 8 apps in tfspreview hosted build	234
Provide check-in/check-out functionality through Team System Web Access	218
Make the “My Work” section available in VS 2012 Professional	215
Add git blame support	212
Kanban board: More customizations and usability features	198
Add ID numbers to the Task Board tasks and work items (PBIs and Bugs)	185
Drag and drop reordering for multiple consecutive items in the product backlog	184
Test Suite Security Permissions in MTM	184
Allow testers to copy test cases by creating new duplicates, not adding references	179
Collapse Product Backlog Items marked as Completed.	170
Allow Test Case Work Items to be created for Native C++ Unit tests	166
Visual Studio Tools for Git Extension (Microsoft Git Provider) integration with Source Control Explorer window	164
Include feature of Task Board Enhancer plug-in	159
Provide more visibility for a test case’s result status	152
Add “tags” support in Git Provider	147
make Test Scribe word reports customizable	143
Allow Bugs to show on the Board	135
See which test plans/runs are running on a lab environment	134
Git Provider should *not* add new files to index automatically	130
Improve Test Manager performance	127
support Spec Explorer in Visual Studio 2012	126
provide a way to track history of test plan and test suites	98
Coded UI Test must support Silverlight 5	95
Fix the ctrl+e shortcut in test runner	85
Allow Git Tools for Visual Studio to Rebase on pull	81
Allow TFS task board to show Tasks without parent	76
By design the board only shows tasks that are associated with Product Backlog Items (it does not show ‘free-floating’ tasks).	76
Allow VSO Stakeholder License to use Feedback Client	75
Visual Studio 2012 coded UI test *must* support Chrome, Safari and FF4!	72
Provide customization for test plan, test results	72
Burndown chart should use the capacity of the team for the ideal trend	71
Improve the usability of the Lab manager component of test manager	71
Drop “Test Explorer” view and revert to the VS 2010 Test View and Test Results paradigm	70
Enable Kanban Column field to be used in work item queries	67
TFS Build: download Git submodules automatically	64
Make it possible to read/write the team capacity with the TFS api	61
coded ui Chrome browser support	57
Parameter passing while queuing builds from TFS Web	54
TFS Scrum: Show state of PBIs and Bugs on board	54
Bring color back to VS 2012 RC Source Control Explorer	51
Provide feedback when unit tests are running	51
Export test plan tree between projects and collections in MTM	51
add real horizontal swim lanes to the Kanban board	50
Support Deployment of Multiple Websites to an Azure Web Role from VSO	45
Add custom fields to allow better Team Build management	42
Make the TFS work item number copy-able	42
Allow cut and paste in Unit Test output window.	38
Add spell check functionality to MTM so that test case writers have the ability to spell check test case steps etc.	38
For TFS 2012 Beta, In scrum template : limit the Assigned To list to my team members instead of showing all existing users	35
Provide TFS Build Source Indexing for Git	34
Support Coded UI Testing for Windows Store applications	34
Provide an audit log of changes to TFS build configurations	32
task reordering	31
Allow using ENTER or SHIFT ENTER when writing test steps	31
Search tests by category in the Unit Test Explorer	29
Provide fine-grained permissions on test plan	27
Better commit message on Visual Studio Online pull request merges	22
Edit test steps within Visual Studio or TFS Web Access	22
Hi, we need the Silverlight plugin related to Coded UI in Visual Studio 2013. Our existing Testscripts testing Silverlight cannot run.	22
Drop MSTest, let any test framework plug into the IDE	21
Create a PowerPivot/Tabular Model store that can be populated from the TFS Service	20
more filteroptions for TestManager Testingcenter	19
Allow Visual Studio Online users with Basic license to view Features list	18
Allow visibility of Test Plan results summary from the Test page in TFS	16
Query based suites in MTM	14
email test case from MTM	13
When sending a work item to Outlook to use Outlook mail format options	12
Ability to copy multiple test cases simultanuously	12
Test Manager 2010: Export Test Cases and preserve the fields/format/parameters	11
Allow the Tester field to be configurable	11
Allow acceptance testers to access TFS test hub	10
Provide a way to filter the list of items in the product backlog (under Work – Backlog) on the web view	10
Provide a licence for non devs that allows Team Room access	10
build security	10
Enable searching through work item query results in Team Foundation Server	10
Make Test and Lab Management available to MSDN Premium Subscribers	10
There must be some description field of Test Suite	10
Merge Test Manager functionalities in Team Explorer / Web Access	9
Support the ability to deploy a test agent to Azure	9
Visual Studio Online (TFS) should show task numbers on task board	7
Reintroduce the shortcut keys for running and debugging the current unit test	7
Allow more than 20 backlog items to show in the “New” column of the Kanban board	6
Allow the tester to modify the test steps in Test Run screen just like QC allows.	6
VSO REST API: provide a way to disable rules for Migration scenarios from other providers.	5
Allow show tags on taks on the sprint board.	5
With VS 2012, give more grouping options for unit tests window.	5
Editing additional fields from the board cards	4
Add the ability to edit on the fly while running a test	4
Ability to collect Custom Logs via MTM test settings.	4
Allow the tester to have options to mark the test case as N/A, Not completed	4
Encode TFS work item queries in a URL	3
Assign/E-mail Stakeholders from Backlog Item	3
In Visual Studio Online, copy the comments with failed tests into the bug report that is generated	3
Improve navigating to links for touch screen in VSO	3
Display backlog items #id in backlog board view	3
Test Manger 2010 customization	3
Add the unit testing wizards back	3
add support for [shift+CR] Linebreaks in MTM Shared Steps, Test Steps, etc.	3
Integration between QTP & Team Foundation Server to support the CI process	3
Add unit testing to express editions	3
TFS Online change order of tasks	2
Set Task Remaining Hours to 0 when Status is set to Closed.	2
Enable copying of test case steps to a new test case	2
Add feature in TFS to export and import test case to and from TFS with steps,actions and expected result details	2
add visual studio online as a category on uservoice	1
Copy and paste test steps from Excel into Test Suite Grid view with alerts if others have modified the existing test cases since Iast pull	1
Moving user stories to new iteration should not change iterations of closed sub tasks	1
Freeze toolbar and states on Kanban Board	1
Allow browsing for WITs when pressing “link to…”	1
View test results in team web access	1
… to enable drag and drop for pbi in feature view. would be useful while assigning pbi to current sprint.	1
change the default config of the TFS2013 Kanban default swimline “Done” to the 20 most recent	1
bring back inline assigned to and effort fields on the task board	1
Inline search in query column headers	1
CodeLens Support For Visual Studio 2013	1
Allow Creation of Work Items w/o a Paid VS Online Subscription	1
Enable the ability to hyperlink to Test Plans from SharePoint	1
Extract parameter list from MTM to Excel/CSV	1
Export MTM 2013 test cases to excel . currently Test scribe or test case extractor not working in 2013	1
Limit limit available users for test points in MTM	1
Would like to have the hyperlink active for authoring test cases	1
perhaps have a license type that allows business users to run (only) test cases.	1
Make it possible to add and query by tags in MTM 2013	1
Display Team Project name in Test Manager	1
I have an issue with MTM: When I run a test and found a bug at step 13, I cannot pause the test,start running a test, following it step by s	1