How we use User Voice to make a better product

Avatar

Ewald

To increase our engagement with our customers, we created the Visual Studio User Voice site to capture your feedback for Team Foundation Server (TFS) and Visual Studio Online (VSO). Since then more than 20,000 people have been participating and we have received over 2,300 suggestions. In the last year we have completed 150 suggestions. We are excited to have so many people who have taken the time to give us feedback. The goal of this post is to describe how we use the User Voice data, and what we have changed to improve our response to the User Voice feedback. If you think we can improve how we use User Voice, please leave a comment or reach out to me via email: ewald.hofman at microsoft.com.

Gaining insight

When we started with User Voice, we received a handful of suggestions per week. Many of these items were already on our backlogs and to be as transparent as possible to our supporters we marked these suggestions as “Under Review”. However we haven’t made any progress on them yet and as a result we haven’t responded to these suggestions for more than 4 years. It may not be a surprise that we created the perception of not listening to customers. We felt really bad about it, so late 2013 we started an initiative to improve how we interact with User Voice and its community.

The first step in the process was to groom the list of User Voice suggestions. We merged duplicates, updated the status of already completed suggestions and assigned the most popular suggestions to the correct feature teams. When we shared the top 20 most popular suggestions, trends and whether we were addressing them for the next major release with all the feature teams, it generated a lot of interest about User Voice. And we didn’t stop with that one time exercise, we continued to groom the User Voice suggestions to keep the list healthy, and regularly we share a summary report to all feature teams working on TFS/VSO.

Giving insight

After we gained the insight in the User Voice data, it was time for the next step. Working with User Voice was a one-way communication: we listened to the community, but did not update the status of completed suggestions nor did we tell you if we were working on something. To improve in this area we made two changes.

  1. Track suggestions on our backlogs: We did not complete the User Voice suggestions, because we did not have a link between our backlog and User Voice. Therefore the feature team was just not aware that a shipped feature was backed by User Voice. To improve that we created bi-directional links between our backlogs and the User Voice for the most popular suggestions. When we ship a feature we can easily find the User Voice suggestions that we need to update. If you see an open suggestion which you think is already completed, please let me know and I will get it fixed.
  2. Frequent updates on in-progress items: The “Under Review” state was overloaded because we used it for so many suggestions in the past. To better indicate which one we are actively working on or which ones are on our 6-month plan, we introduced the states “Started” and “Planned”. The goal is to give an update on in progress suggestions at least every 3 months. The majority of the Under Review suggestions are still on our backlog, but they are not part of the 6-month plan. We are going to bite the bullet and reset the state of all these Under Review suggestions, to make it very simple to you. If a suggestion has a status, you can expect us to give regular updates until we have completed the item.

We are listening…

To prioritize our backlogs we use many channels for input, and User Voice is one of these channels. And we take it very seriously.

  • Below you see the list of 150 suggestions that we have completed in the last year. Based the User Voice feedback we completed items which were not in our original plans, including the “Creating Queries – Token for Current Iteration” and “ability to add multiple activities per team member in sprint”. But we heard you loud and clear and we prioritized these items higher on our backlogs.
  • There are currently 49 items in progress (state is “Planned” or “Started”). Some of them have gone stale, but in the next few weeks all in progress suggestion will have an recent update. You can hold us accountable for that, so if you see an in-progress suggestion without a recent update don’t hesitate to reach out to me.
  • Recently Brian Harry has published a blog post which shows some of the items that we are planning to work on in the next few months, and that list is influenced by User Voice.
  • In each planning cycle the feature teams are being questioned why they are not working on one of the top 10 items

These are the suggestions that we have closed in the last year sorted on the number of votes.

SuggestionVotes
rename project in TFS6159
Add “Create Unit Tests” menu option back!3038
Creating Queries – Token for Current Iteration2188
Make Code Review available to Professional users1221
Customization of TFS Web Access1201
ability to add multiple activities per team member in sprint1094
Allow TFS Task Board to have custom colors880
In the team Explorer, Change Default State value of a selected Work Item to “Associate” instead of “Resolve” on Check In845
Excel integration with MTM792
Allow tests to be grouped by class, category, and namespace in Test Explorer699
Support more browsers and browser versions in MTM520
Integrate Team Wiki / project documentation pages471
Team Test – Improve the usability of Microsoft Test Manager457
Bring back the view all pending changes window (VS11)436
Use a build controller over more than one Team Project Collections397
Provide the ability to prioritise Backlog items in a sprint backlog by dragging and dropping the item340
Provide TF build activities like “deploy web app”, “deploy database”, or “copy file”305
Allow tests to be paused, allow other tests to be run, and then resume the paused test case293
Suite created by “Workitem and direct links” query in MTM282
Implement TFS Build for linux (Create Linux build agent)272
Continue support for Database Unit Tests in VS 2012267
Testers should be able to get parameters for a test case from a csv file260
Show me all test suites/ test plans a test case belongs to245
Provide versioning of test cases to support multiple parallel releases243
Allow testers to edit test cases inside Manual Test Runner241
Support Windows 8 apps in tfspreview hosted build234
Provide check-in/check-out functionality through Team System Web Access218
Make the “My Work” section available in VS 2012 Professional215
Add git blame support212
Kanban board: More customizations and usability features198
Add ID numbers to the Task Board tasks and work items (PBIs and Bugs)185
Drag and drop reordering for multiple consecutive items in the product backlog184
Test Suite Security Permissions in MTM184
Allow testers to copy test cases by creating new duplicates, not adding references179
Collapse Product Backlog Items marked as Completed.170
Allow Test Case Work Items to be created for Native C++ Unit tests166
Visual Studio Tools for Git Extension (Microsoft Git Provider) integration with Source Control Explorer window164
Include feature of Task Board Enhancer plug-in159
Provide more visibility for a test case’s result status152
Add “tags” support in Git Provider147
make Test Scribe word reports customizable143
Allow Bugs to show on the Board135
See which test plans/runs are running on a lab environment134
Git Provider should *not* add new files to index automatically130
Improve Test Manager performance127
support Spec Explorer in Visual Studio 2012126
provide a way to track history of test plan and test suites98
Coded UI Test must support Silverlight 595
Fix the ctrl+e shortcut in test runner85
Allow Git Tools for Visual Studio to Rebase on pull81
Allow TFS task board to show Tasks without parent76
By design the board only shows tasks that are associated with Product Backlog Items (it does not show ‘free-floating’ tasks).76
Allow VSO Stakeholder License to use Feedback Client75
Visual Studio 2012 coded UI test *must* support Chrome, Safari and FF4!72
Provide customization for test plan, test results72
Burndown chart should use the capacity of the team for the ideal trend71
Improve the usability of the Lab manager component of test manager71
Drop “Test Explorer” view and revert to the VS 2010 Test View and Test Results paradigm70
Enable Kanban Column field to be used in work item queries67
TFS Build: download Git submodules automatically64
Make it possible to read/write the team capacity with the TFS api61
coded ui Chrome browser support57
Parameter passing while queuing builds from TFS Web54
TFS Scrum: Show state of PBIs and Bugs on board54
Bring color back to VS 2012 RC Source Control Explorer51
Provide feedback when unit tests are running51
Export test plan tree between projects and collections in MTM51
add real horizontal swim lanes to the Kanban board50
Support Deployment of Multiple Websites to an Azure Web Role from VSO45
Add custom fields to allow better Team Build management42
Make the TFS work item number copy-able42
Allow cut and paste in Unit Test output window.38
Add spell check functionality to MTM so that test case writers have the ability to spell check test case steps etc.38
For TFS 2012 Beta, In scrum template : limit the Assigned To list to my team members instead of showing all existing users35
Provide TFS Build Source Indexing for Git34
Support Coded UI Testing for Windows Store applications34
Provide an audit log of changes to TFS build configurations32
task reordering31
Allow using ENTER or SHIFT ENTER when writing test steps31
Search tests by category in the Unit Test Explorer29
Provide fine-grained permissions on test plan27
Better commit message on Visual Studio Online pull request merges22
Edit test steps within Visual Studio or TFS Web Access22
Hi, we need the Silverlight plugin related to Coded UI in Visual Studio 2013. Our existing Testscripts testing Silverlight cannot run.22
Drop MSTest, let any test framework plug into the IDE21
Create a PowerPivot/Tabular Model store that can be populated from the TFS Service20
more filteroptions for TestManager Testingcenter19
Allow Visual Studio Online users with Basic license to view Features list18
Allow visibility of Test Plan results summary from the Test page in TFS16
Query based suites in MTM14
email test case from MTM13
When sending a work item to Outlook to use Outlook mail format options12
Ability to copy multiple test cases simultanuously12
Test Manager 2010: Export Test Cases and preserve the fields/format/parameters11
Allow the Tester field to be configurable11
Allow acceptance testers to access TFS test hub10
Provide a way to filter the list of items in the product backlog (under Work – Backlog) on the web view10
Provide a licence for non devs that allows Team Room access10
build security10
Enable searching through work item query results in Team Foundation Server10
Make Test and Lab Management available to MSDN Premium Subscribers10
There must be some description field of Test Suite10
Merge Test Manager functionalities in Team Explorer / Web Access9
Support the ability to deploy a test agent to Azure9
Visual Studio Online (TFS) should show task numbers on task board7
Reintroduce the shortcut keys for running and debugging the current unit test7
Allow more than 20 backlog items to show in the “New” column of the Kanban board6
Allow the tester to modify the test steps in Test Run screen just like QC allows.6
VSO REST API: provide a way to disable rules for Migration scenarios from other providers.5
Allow show tags on taks on the sprint board.5
With VS 2012, give more grouping options for unit tests window.5
Editing additional fields from the board cards4
Add the ability to edit on the fly while running a test4
Ability to collect Custom Logs via MTM test settings.4
Allow the tester to have options to mark the test case as N/A, Not completed4
Encode TFS work item queries in a URL3
Assign/E-mail Stakeholders from Backlog Item3
In Visual Studio Online, copy the comments with failed tests into the bug report that is generated3
Improve navigating to links for touch screen in VSO3
Display backlog items #id in backlog board view3
Test Manger 2010 customization3
Add the unit testing wizards back3
add support for [shift+CR] Linebreaks in MTM Shared Steps, Test Steps, etc.3
Integration between QTP & Team Foundation Server to support the CI process3
Add unit testing to express editions3
TFS Online change order of tasks2
Set Task Remaining Hours to 0 when Status is set to Closed.2
Enable copying of test case steps to a new test case2
Add feature in TFS to export and import test case to and from TFS with steps,actions and expected result details2
add visual studio online as a category on uservoice1
Copy and paste test steps from Excel into Test Suite Grid view with alerts if others have modified the existing test cases since Iast pull1
Moving user stories to new iteration should not change iterations of closed sub tasks1
Freeze toolbar and states on Kanban Board1
Allow browsing for WITs when pressing “link to…”1
View test results in team web access1
… to enable drag and drop for pbi in feature view. would be useful while assigning pbi to current sprint.1
change the default config of the TFS2013 Kanban default swimline “Done” to the 20 most recent1
bring back inline assigned to and effort fields on the task board1
Inline search in query column headers1
CodeLens Support For Visual Studio 20131
Allow Creation of Work Items w/o a Paid VS Online Subscription1
Enable the ability to hyperlink to Test Plans from SharePoint1
Extract parameter list from MTM to Excel/CSV1
Export MTM 2013 test cases to excel . currently Test scribe or test case extractor not working in 20131
Limit limit available users for test points in MTM1
Would like to have the hyperlink active for authoring test cases1
perhaps have a license type that allows business users to run (only) test cases.1
Make it possible to add and query by tags in MTM 20131
Display Team Project name in Test Manager1
I have an issue with MTM: When I run a test and found a bug at step 13, I cannot pause the test,start running a test, following it step by s1

0 comments

Comments are closed.