This is special guest post from Mark Grinols, a development lead on the Minecraft team at Microsoft.
A key aspect of Minecraft that players love is the ability to play with family and friends on nearly any platform they choose. Offering this breadth of choice to hundreds of millions of players worldwide is a substantial engineering challenge. In addition to the original version in Java on Windows and macOS, we’ve added cross-platform functionality with the C++ codebase using the Bedrock Engine, which runs on multiple form factors and OSes, including mobile (iOS, Android and Windows Mobile), consoles (Xbox and Switch), set top boxes (Fire TV and Apple TV), AR/VR platforms (GearVR, Oculus Rift and HoloLens), and desktops (MacOS and Windows).
While our players delight in the platform breadth, behind the scenes the game team must carefully craft user experiences to work great on screens from tiny to giant, and that work intuitively whether via gamepad, touch, mouse and keyboard, or VR gesture. Engineers must determine how each new capability will be supported on each of our platforms while factoring in the idiosyncrasies of an individual platform’s OS and ecosystem. To make this possible, the test automation system must ensure that in the end it all works (and keeps on working) on every platform.
Integrating App Center into our Cross-Platform Engineering Workflow
We make extensive use of cloud services to solve the challenges inherent in a cross-platform game built by contributors from five game studios around the world. We use Github for source control, Visual Studio Team Services (VSTS) for work item management, build systems, and test results, and Azure for build agents, storage, web services, and more. We depend on App Center for running test automation and collecting player crash telemetry for mobile platforms.
Let’s walk through the workflow of getting code submitted to the game. It begins with a developer pushing their topic branch to GitHub and opening a Pull Request (PR). In response, VSTS triggers roughly 30 different builds covering a breadth of platforms, architectures, and build-flavors.
When game builds complete, VSTS kicks off “TestRunner” builds that orchestrate test automation. If the platform is iOS or Android, a test run queues in App Center Test. For other top platforms, tests execute in our on-premise test labs. Our tests are a mixture of cross-platform unit tests and a suite of integration tests that drive the game through key scenarios. When runs complete, results are posted to our Test Results service in Azure. We put it in the cloud so the same results and reporting processes run regardless of where the tests were executed. Test run details are then piped back into VSTS, while results summaries are sent to GitHub as commit statuses. This way, the developer gets a clear thumbs-up or down on automation results right on their Pull Request page, and they can click through to details and logs in VSTS. If the developer updates their topic branch, the whole process begins again. They are only allowed to merge the PR to the main branch after results are green and reviews complete.
Figure 1: Minecraft’s Pull Request -> Test Automation workflow
Continuous Integration and other workflows operate similarly. All told, we create around 3,000 builds and 700 test runs per day. App Center Test is a crucial piece of the puzzle, providing the test execution environment for iOS and Android, two giant platforms for the game.
The flexibility of the App Center service and SDK made it surprisingly easy to integrate into this workflow. We’d expect it to be similarly straightforward with TeamCity or another CI system. Android and iOS both being supported means the integration effort was amortized across two platforms, and as new App Center platforms come online, we hope to leverage them as well.
While we’ve lost a little bit of control and direct device access, it’s been worth it for the benefits of device breadth far beyond what we could achieve on our own, on-demand device availability, non-virtual hardware we can run perf tests on, and a stable, reliable test service that lets us focus more on building a great game and less on our infrastructure.
App Center and Our Write-Once, Run-Everywhere Test Automation Strategy
Our goal is to have automated unit tests and integrations tests that run on real devices for every platform we support. Typical test frameworks support a tiny subset of our platforms list. We could have compromised on platform breadth and adopted a single test framework. Alternatively, we could have integrated and written tests for as many frameworks as needed to cover all the platforms (paying the giant maintenance cost forever). Not happy with either choice, we implemented a custom test framework inside the game itself. Tests are written in C++, in the same codebase by the same developers who work on features and bugs. Every test can run on every platform the game does.
| Pros | Cons |
|---|---|
|
|
|
|
|
|
|
|
|
How do App Center tests fit into this picture? Let’s drill into the flow:
Figure 2: Invoking a test run in App Center.
Interaction with App Center begins with the build agent using the App Center command line interface (appcenter-cli) to ask the service to kick off the test run on the appropriate platform (iOS or Android). We have a single, simple C# Xamarin UITest test case that launches the game and tells it what tests to run. This communication happens via backdoor methods. Minecraft Bedrock is 99% common C++ code, but for each platform there is a small amount of ‘native’ code for platform integration; Java on Android and Objective-C on Apple platforms. This is where backdoors are implemented, following the protocol defined by the App Center SDKs.
Figure 3: A Minecraft test running in App Center.
Once the game has been told what tests to run, it takes over. While Minecraft sees the test run as including any number of tests, from App Center’s perspective it’s just one test. Here’s what our App Center Test Run dashboard might look like at a point in time:
Once test execution is complete, the game posts results to our Test Results service in Azure, where it gets analyzed and reported upon. This flow is consistent across all platform and execution environments, which simplifies everything. Our usage pattern may not be typical, but the App Center platform gives us the flexibility we need to structure things in a way that works for our game.
We also leverage App Center in production crash telemetry. We’ve used HockeyApp for Android and iOS crash analytics for several years. Integrating the SDKs was simple, and their Deploy to HockeyApp VSTS build action makes registering symbols seamless.
HockeyApp reports on crashes that end users are hitting in our game, collated by platform, version, and call stack. This lets us easily see our top issues for a given release. When specific crashes reach a certain hit count, bugs are automatically opened in our VSTS bug database.
Unfortunately, the typical HockeyApp crash can’t be easily reproduced in-house (otherwise we’d probably have found it before releasing). Knowing how many users are hitting it across build and device type is data we need to triage and isolate the bug. We have nearly 1.5 million community members signed up for Minecraft betas at any given moment, and the data we get from HockeyApp is a key input to deciding when we’re ready to release a build more broadly. HockeyApp’s unified view of this data is easier to use than the wildly different capabilities and presentation you get looking at platform vendors’ dashboards (Google Play Console, Apple iTunes Connect, Windows Dev Center, etc).
We’re currently adding HockeyApp crash telemetry support for our Minecraft Realms dedicated server service (which runs on Linux). We plan to upgrade all of our platforms to use the App Center SDK (the next generation of HockeyApp) as soon as we can.
In Closing
We’re committed to the broadest possible platform support in Minecraft—few games have as much breadth—and all of our engineering systems are built with this in mind. App Center lets us run tests on a huge diversity of always-available Android and iOS devices (not simulators or emulators), and the flexibility we needed to plug it into our overall workflow seamlessly. HockeyApp gives us critical crash data on mobile we use every day. The takeaway for me is that App Center need not be the center of your workflow. You can pick and choose the parts of App Center that best solve your needs, without worrying about the rest. Check out the services App Center offers and see which ones bring value to your pipeline and workflow.
About the Author
Mark Grinols is a Development Lead on the Minecraft team in Redmond, WA. He’s spent much of the last 20 years focusing on automation systems on teams all around Microsoft. He loves figuring out sustainable ways to deliver better software faster.
