Hello. This is Pengpeng again.  I wrote a blog not long ago talking about security testing for the Visual C++ IDE as a parser. The IDE is definitely a lot more than just a parser. Among other things, it provides an integrated software development experience with browsing, go to definition, go to declaration, quick info, auto completion … There is just so much you can do with the UI.

To guarantee great features with great quality, as testers, we have to be innovative and customer focused. This implies that we need to test a lot of end-to-end user scenarios. For instance, a user creates some new Visual C++ projects, defines new classes and functions, declares and instantiates new objects and makes function calls. During this process, for correctness, she/he has to reference the class/function definitions using go to definition, or class view, or quick info, or code definition window … Consequently, this makes the tests really complicated and calls for innovations in our testing infrastructures. Today, I am going to talk about one such innovation called “Build On The Fly (BOTF)”, which changes the way we automate our tests from building and deploying the test binaries all at one time to, as the name implies, on the fly.

A little bit of background on our current test execution architecture. It comprises a driver program and the test codes. The test codes for a testcase are: the test binary, a .dll file, and the test metadata given in an xml file. The metadata also provides the test execution information such as the entry point of a testcase in the .dll. When a test executes, the driver is called with the test xml file as the parameter. The driver reads the xml, find the .dll test binary specified in the xml file, drives the test according to the test data in the xml file, and log appropriately the test results.

Before the era of BOTF, the test binaries were generated daily and aggregated into several big msi’s using a dedicated build machine. During test automation, the msi’s were first copied to, and then installed, on the test machines. As a result, all the test .xml files and .dll files were generated in advance. This approach becomes really painfully: besides the maintenance of the build machine, often times, test issues are convoluted with build issues, and it comes really complicated to figure out what exactly went wrong during test failures. In addition, it brings a lot of overhead: any test issue fix has to wait for the daily build to be ready, and to be deployed, in order to verify.

BOTF solves the problem by building the test binaries on demand. First, in the test metadata, the xml files, the tests are labeled as BOTF. During the setup stages, the raw test codes, the .vcproj, .h and .cpp files are copied to the test machines. The test execution process is exactly as before: the driver is called with the test xml filename as the parameter. Underlying, now the driver will first build the needed test binary and then execute it. With this innovation, all the tests self contained: build or test issue can be easily isolated. All the test fixes are also “on the fly” and we do not need to wait for the build machine any more.

Thanks.

Pengpeng