This is a special guest post from Marco Kuiper, Consultant at Info Support and blogger at Marcofolio.net.   Many developers are good at building things right, but far fewer are skilled at building the right thing. While you need a strong idea of what problem you’re solving for your users (and why and how), you can’t rely on vision and gut feeling alone. Gathering insights and feedback from your users is critical to winning their loyalty, as well as to your long-term success and profitability as an app developer.

In my work as a professional developer, I’ve learned how to couple quantitative information with qualitative user feedback to ensure my team delivers the best possible user experience.

To help you do the same, I’ll share:

• How to use Visual Studio App Center to gather the necessary user and app behavior data required to make informed product decisions.
• A few of my favorite ways to collect less structured insights.
• Why both types of information are important and help you build better apps.
• Simple tips to apply what you learn from metrics and user feedback, and start improving your apps.

At Info Support, we help organizations develop, manage, and iterate on web, mobile, and software solutions, as well as advise on strategy and the latest development techniques and best practices. Our clients range from the agriculture and healthcare sectors to public mobility and financial services.

I’ll focus on B2B apps (defined as business to business, built for and distributed to employees via an enterprise store), since many of my projects target business users. While many of these principles apply to B2C apps (business to consumer, where apps are distributed publicly) too, there are also a few distinct nuances, namely how you distribute, market, and respond to user reviews.

The Importance of Getting Feedback Fast

LeSS uses a great piano analogy to reiterate the importance of getting feedback fast. If you’re learning to play a piece on the piano, you hear a key’s note immediately after hitting it. You get feedback immediately, and if something is wrong, you correct it. You learn fast, iterate, and keep improving.

But, what if your piano didn’t play your note back until a week later? Learning to play the piece would take significantly longer, because you’d be in the dark about any mistakes. When you delay feedback, it takes longer to learn what’s working and what isn’t.

The same is true for you as you build new features and functionality in your apps. If you get feedback as fast as possible, you’ll learn and adapt quickly. The longer you wait, the more code you add, and it’s significantly harder and more expensive to course-correct.

Beta User Groups and Feedback Collection in 6 Simple Steps

Step 1: Set Up Your Beta Groups and Select Users

Feedback is important and beta users are your secret weapon to getting the insight you need. Listen carefully and be receptive to what they say, since they have a unique perspective that might be more accurate than your development team’s; as hard as it is to hear this, it’s usually true!

One of my favorite tips: create a beta group of users who feel responsible and invested in the product’s success, like team members who are your app’s ultimate end users. Since their feedback will directly impact their future experience, they’re likely to be more responsive and honest.

Bonus points: select a good cross-section of your total user population (not just across devices and locations, but characteristics like age and familiarity with mobile technology). From a size perspective, bigger isn’t always better. We’ve had great success with 30-100 active beta users providing feedback for apps with a user base of 10,000+.

How do you find beta users? Simple: ask. Send an open invitation to everyone at the company, asking if they’d like early access and have an impact on your app’s design and UX.

If you’re a B2C app, it might seem hard to find the right beta testing profile, but the same general principles apply. Again, simply ask your users if they’d like to get early access to new features and directly impact your roadmap. You’ll be surprised at how many people are willing to help.

Our definition of beta users (or testers) is a designated subset of your user population who receive early access to new features and directly communicate with the development team. Treat them like extended team members or friends, willing to spend time and energy providing feedback to help you make your apps better.

Final tip: In communication, we call these users Ambassadors, not beta users or testers, to stress their importance to us and the role they play in our success. You could use another engaging, positive term, such as Power Users or Head Starters.

Step Two: Invite Testers into Early Development Conversations

My team uses Scrum, and we invite our beta members to join our Sprint Reviews for a few reasons:

• Showing beta users new functionality in real-time gives our development team the direct and immediate feedback we’re after.
• Sharing what we’re planning in the upcoming iteration often leads to questions or reactions that steer what we build (or don’t build) next.
• Our beta users feel engaged, since they know we’re making an effort to get their feedback throughout the development process, and we’re truly listening.

Step Three: Get Test Builds to Users as Quickly as Possible

This one’s easy. We suggest setting up App Center Distribution Groups to automatically deploy app updates to your beta testers, notify them about new builds, and make sure they always have the latest version.

Step Four: Establish Two-Way Communication

After you get your builds into users’ hands, your testers need a place to share their thoughts, feedback, and problems.

We use Yammer to interact directly; beta users post comments, upload files, @ mention relevant team members, and start threaded conversations at any time.

Invite your beta testers to an online communication platform, and, trust me, magic will happen: users provide feedback and communicate directly to the team (great!), and they’ll interact with each other (better than great—magic!). Sharing ideas and “listening” to conversations between users is something we never expected, but it’s led to making many of our projects hugely successful.

Step Five: Be Responsive and Separate Signals from Noise (Prioritization!)

Be prepared for broad feedback and establish a plan for acting on user commentary, ranging from new ideas (we add these to the Product Owner’s backlog), information that invalidates your assumptions or hypotheses, or compliments about your work (we love when we hear that we’re improving users’ workflows!).

Our Ops team plays a critical role in monitoring our Yammer networks, responding to questions, and helping users. When they a frequent request or identify an unsolved need, we discuss in our Sprint Reviews, prioritize, and add to our backlog.

Step Six: Plan for Production User Feedback

Beta users are invaluable, but don’t forget your end users. It’s important to make it easy for your production users to provide feedback, too. We invite users to provide app reviews via Zendesk, creating a community for each app / business, and inviting users to share comments or report bugs.

We have a dedicated support team to help users with questions and provide troubleshooting tips, so we don’t always respond directly, but we monitor these conversations to see production users’ feedback.

As an added, and unexpected, benefit, beta users often hop into discussions, responding to colleagues’ questions, and offering proactive support when they see colleagues using our apps in the field.

Beyond Beta: Use Feature Flags to Experiment and to Expedite Learning

Ready to supercharge what you learn from your users? Meet feature flags.

With feature flags, we deploy the same app to everyone, but enable/disable certain features dynamically for specific groups of people. Dynamically changing our app allows us to get feedback about specific features and helps us understand where we need to improve. Based on what we’re trying to accomplish with the feature, we can compare users’ reactions and gauge whether we roll it out to all users.

Building feature flags isn’t difficult, and there are several implementation methods. We love statically typed feature flags (like this one on GitHub), because they allow us to easily test functionality with or without feature flags enabled.

To implement feature flags in your apps:

• During app start up, retrieve flag state to see what functionality you can access and flag. This allows you to dynamically change how the app behaves after start up completes.
• Configure your backend to select a flag state based on a user identifier (such as e-mail address, employee number, user role, or geolocation).
• Decide what functionality you’d like to flag. The easiest option is to show / hide a certain UI element, while changing service call endpoints is more advanced.
• Configure your backend to release flagged features to certain users.

We use feature flags for both beta and production users, releasing different functionalities to subsets of users. For example, we may have one beta group, but we’ll feature flag a feature for 10% of users.

App Center Distribution Groups simplify deploying specific builds (e.g. our test, feature flagged, or general availability versions) to specific user groups, and, to truly achieve Continuous Delivery, we separate the deployment and release process.

• Deploying is both a technical and development team decision. Deploying is something that should have a certain cadence, like updating your app every two weeks (Continuous Delivery).
• Releasing is a joint business decision. Your business stakeholders, Product Managers, and technical team decide when to ship new functionality to users. This doesn’t have a certain cadence, but happens when your update is “ready or needed.”

Because we use feature flags, we build functionality and deploy, but hide from the user. When everyone decides it’s time to release, we simply enable the feature for all users.

Since we enable flags at start up, users see the new feature the next time they launch the app, with no updates required.

When to Feature Flag: Sample Scenarios

Beta users: We fully test all new features, but releasing to production can be tense. Beta users to the rescue! We call a few testers, request they quickly check out a new feature, enable a feature flag, and distribute the updated version to their devices. Once they confirm it works, we’re confident and roll it out.

Production users and launches: Feature flags give us the flexibility to slowly push updates to all users, which is especially useful if you have a large user base, or want to verify that your backend can handle the additional traffic your new feature generates.

If you notice lag, or find a bug in your code, disabling a flag is quick and painless. Users don’t need to install an update; the “flagged” feature is just disabled in their existing app.

Feature Flagging Best Practices

When you’re getting started, it’s hard to know what to flag and when, but a few things to remember:

• Your code must be robust and well tested. Feature flags add complexity, so unit and UI tests become even more important.
• Your feature flags shouldn’t conflict. If your flags conflict, you run the risk of confusing your users, degrading the user experience, and introducing bugs. Feature flagging several features is tempting, but also makes it more difficult to determine if you’re seeing real results, or the result of users bumping into conflicts or issues with your code.
• Note that feature flagging and separating deployment from release requires all users to have the latest version of your app installed (or the version where you’ve included the feature flag you’d like to roll out). Users with older versions won’t see the new functionality, even if you enable the feature for all users.

If you’re a React Native developer, App Center’s CodePush service allows you to use feature flags to A/B test and experiment with different implementations. You create two (or more versions), push updates to users instantly, log anonymous responses and user engagement, and can return to “normal” at any time. Check out John Wargo’s blog post to get started.

Numbers and Metrics Matter, Too

Now that we think we’re heading in the right direction, we need numbers to back up our beta feedback and feature flag results.

Using Analytics, especially Event Metrics, we’re able to learn more about the behavior of our app and our users (as well as user demographics). Are people actually using the new functionality? If so, how? Did the change we pushed affect user engagement or time spent in app?

Several factors shape our decisions to move functionality to production. Beyond the obvious, “it works as expected,” feature flags help us compare and contrast how new features affect user experience.

To do this, we:

• Give any new feature a unique name, so it’s easier to find and analyze logged data.
• Use Application Insights to drill into usage (time in app, sessions, etc.).
• Extrapolate beta tester (or feature flagged group) data to see how the feature will affect all production users.
• Measure against our goals. For most B2B apps, we aim to make users more efficient, increasing worker productivity, and helping them get back to work faster. If a new feature causes users to spend less time in our apps, we’re on the right track.

Ultimately, the product owner makes the final decision on what we do next. He or she answers the hard questions: should we continue our efforts or take a different route? But, with both quantitative and qualitative information at their disposal, our product owners are confident that they’re guiding the team in the right direction.

Final Thoughts

To build the right thing, take both your beta and production user feedback seriously, and back up it up with data from a service like App Center’s Analytics.

Balancing your apps’ metrics, engagement stats, and user feedback is never an easy task (check out one of my favorite blogs on the topic). The more you know, the easier it’ll be to make decisions.

With numbers and opinions, you’re more likely to discover that something you think is a major issue doesn’t impact your users, or—more likely—the reverse. Your reports might not pick up something that’s a frequent point of contention in user forums or with your beta testers.

Once you nail the quantitative-qualitative combination, you’re on your way to working on apps that are fun to build and that beloved by your users.     If you haven’t already, create your Visual Studio App Center account, connect your first app, and start shipping better apps now.

Marco Kuiper is a Consultant at Info Support, blogging on Marcofolio.net, frequent speaker and is always incurably optimistic. He inspires and motivates people to share and expand their development knowledge.