Servicing: The History of Visual Studio… Today!
Hello everyone, my name is Jamie Eckman. I am a Software Design Engineer in Test (or SDET, for short) on the Visual C++ Libraries team, and have been for the last two and a half years. Usually I’m in charge of testing ATL Server and our new STL/CLR library, but recently I’ve been working on the subject of today’s post, servicing.
While we would certainly love it if everyone rushed out and bought the latest release of Visual Studio as soon as it shipped, we realize that this doesn’t make sense for many of our customers. What this means, of course, is that we have customers using older versions of Visual Studio. We are committed to helping these customers (and their customers!) for as long as the products they use are still supported. This support can extend for quite a few years beyond the original ship date.
One of the ways in which we support these customers is to verify that no change Microsoft makes will break any functionality in the versions of Visual Studio still in support. And there are a number of Microsoft technologies which directly or indirectly impact Visual Studio: Windows, Office, SQL, IIS, and of course Visual Studio itself. Every time a new version of these technologies is released, and even when some of them are patched (notably Windows), it’s our job to make sure that Visual Studio, and any applications built with Visual Studio, still work.
As Ben Anderson touched on a couple months ago, the primary vehicle of this testing is our automation. There is no other way to deal with the amount of testing we have to do. But which tests do we run? Tests change and new ones are created as technologies evolve. If we ran the tests for Visual Studio 2005 against Visual Studio .NET, most of them would probably fail. The solution is to run the tests as they existed when Visual Studio .NET was release.
Conveniently, we still have all of those tests hanging around in their own branch of our source control tree. One of QA’s first jobs after a new version of Visual Studio is released is to branch all of our tests and verify that they can run in our sustained engineering lab (as opposed to our own lab). Here is where we find out which test authors have been naughty and writing tests with dependencies outside of the test machine. The sustained engineering’s test lab generally doesn’t have access to the same machines our lab does, so if I write a test that depends on the FTP service running on my main work machine, it will fail in the Sustained Engineering lab.
In addition to fixing these problems we disable any test that fails because of bugs in Visual Studio. It’s regrettable, but yes, Visual Studio ships with known bugs (although we do try to ensure that these don’t affect core scenarios). Since the purpose of service testing is to look for regressions, tests which fail from the outset are not useful and simply cause noise in the test results.
Once all of these issues are taken care of, our tests are handed off to sustained engineering, and from that point on they own them. Whenever a new patch for Windows or a new version of Office is about to be released, sustained engineering will run our tests and look for any regressions. Of course, running all of our tests is a bit of a sledgehammer. For low-risk updates to a peripheral technology, running everything is probably overkill. Fortunately, as Ben Anderson wrote, our tests are prioritized, and for the cases where a lighter touch will suffice we can simply run our tests most likely to catch failures.
So there you have it. The history of Visual Studio exists today in our sustained engineering lab, which works hard to ensure that all of our customers, not just those using the latest version of Visual Studio, can provide their customers with great applications and services.
Light
Dark
0 comments