{"id":13145,"date":"2017-06-28T09:31:10","date_gmt":"2017-06-28T14:31:10","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/?p=13145"},"modified":"2019-02-27T06:26:22","modified_gmt":"2019-02-27T06:26:22","slug":"testing-in-a-cloud-delivery-cadence","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/testing-in-a-cloud-delivery-cadence\/","title":{"rendered":"How we approach testing VSTS to enable continuous delivery"},"content":{"rendered":"

I like to write, from time to time about our experiences, successes, failures and learnings from delivering Visual Studio Team Services (VSTS), a large scale service, on a cloud delivery cadence.\u00a0 My most recent post reflected on how cool it is to be able to deliver customer fixes within a day or two<\/a>.\u00a0 And I’ve written many times about our practice of delivering all our work to production every sprint (or, in some cases, even more often).<\/p>\n

Usually my posts are sparked by something I see happen.\u00a0 Today I got an email about progress on our efforts towards reliable tests and it made me think about sharing it.<\/p>\n

When we first started the journey towards accelerated delivery we began by accelerating all the processes we’d had in place for years.\u00a0 It quickly became apparent that doing that would never get us to where we wanted to be and we had to be prepared to do things very differently.\u00a0 In the intervening 6 years or so, we have gone through many transformations – how we plan, track progress, deploy, manage feedback, monitor, architect, develop and test.\u00a0 Change has been constant over the 6 years and we’re no where near finished – I’m not sure we’ll ever finish.\u00a0 If we’d tried to do it all at once. I’m sure we would have failed.\u00a0 Taking it one or two key practices at a time has worked out well for us and allowed us to bring the team and the code base along for the ride.\nA little over 2 years ago, we realized that one of our biggest remaining impediments to our goal of “continuous delivery” was test – everything about test: our org, roles, frameworks, tests, harnesses, analysis, …<\/p>\n

Two years ago, we had 10’s of thousands of tests.\u00a0 They were written by “testers” to test code written by “developers”.\u00a0 While there were some advantages of this model – like clearly measurable and controllable investment in test, expertise and career growth in the testing discipline, etc. there were also many disadvantages – lack of accountability on the developers, slow feedback cycle (introduce bug, find bug, fix bug), developers had little motivation to make their code “testable”, divergence between code architecture and test architecture made refactoring and pivoting very hard\/expensive, and more.<\/p>\n

A very high percentage of our tests were end-to-end functional “integration tests”.\u00a0 Often they automated UI or command line interfaces.\u00a0 This meant that they were very fragile to small\/cosmetic changes and were very slow to run.\u00a0 Because UI code isn’t really designed to be testable there were often random timing issues and the test code was littered with “Sleep(5000)” to wait for the UI to reach a steady state.\u00a0 Not only was that incredibly fragile (sometimes the UI would take a while – network hiccup or something), it also contributed greatly to the tests taking a very long time to run.<\/p>\n

The result of all of this is that full testing would take the better part of a day to run, many more hours to “analyze the results” to identify false failures and days or weeks to repair all the tests that were broken due to some legitimate change the in the product.<\/p>\n

So, 2 years ago, we started on a path to completely redo testing.\u00a0 We combined the dev and test orgs into a consolidating “engineering” org.\u00a0 For the most part, we eliminated the distinction between people who code and people who test.\u00a0 That’s not to say every person does an identical amount of each, but every person does some of everything and is accountable for the quality of what they produce.\u00a0 We also set out to completely throw away our 10’s of thousands of tests that took 8 years to create and replace them with new tests that were done completely differently.<\/p>\n

We knew we needed to\u00a0reduce our reliance on\u00a0fragile, slow, expensive UI automation tests.\u00a0 We created a taxonomy to help us think about different “kinds” of tests:<\/p>\n