Moving Test Hooks Outside Your Product’s Source Code

 Hello, my name is Raul Perez, I’m part of the Visual C++ IDE team and I’m going to talk about test hooks. When writing automated testing for specific features in your applications you usually have to figure out how you’ll actually get the information you need without going through all the steps you would usually through the user interface (this is what we call component level testing). This is where test hooks come into play; I’m going to discuss the traditional approach versus a more unconventional approach that uses test hooks that live outside your actual application.

What is a test hook?

I won’t go into a lengthy explanation on test hooks since there’s extensive resources about this (Sara Ford had an excellent blog entry about this topic http://blogs.msdn.com/saraford/archive/2005/03/16/396891.aspx). Quoting from there:

“Test hooks are “hidden” messages the developer will write into the application.  I call these hidden because you have to ask the developer what messages to send to which window (or look at the code). The tester supplies the developer with a list of requirements, like being able to press the button or get the caption.  And the developer in returns gives you back a set of messages to send to a given window that will produce the desired effect, like click the button or return the button’s caption.”

Traditional test hooks

As mentioned above test hooks are usually written by developers after the code for the actual feature is complete to open a backdoor to get information out of that specific feature. Since they are indeed a backdoor they live inside the code and are completely dependent on the owners of those codebases.

Consuming test hooks

This topic in itself could be a complete article on its own so I’ll stick to the basics here. Every company (and even different teams inside the same company) use different frameworks that consume these test hooks. For example, on my previous job we had one application that had multiple controls which allowed the users to select some parameters to query the database and then created a report based on the results. To test the database queries I wrote an external application that loaded the assembly containing the controls and using the hooks inside that assembly it created all the possible reports from specific categories taking in either random parameters or parameters I selected.

On the IDE team we have several libraries that we use depending on the features we’re currently testing and when new features come in we modify these to consume the newly created test hooks and write tests that cover a wide range of scenarios. These go from common user scenarios to completely nitpicky cases; I mean we all write code and from time to time look into the weirdest way of achieving something and we expect things to work for all possible scenarios right?

Writing test hooks that live outside the product

What if instead of writing additional code inside the product to could use existing API’s and achieve the same result? This sounds almost too good to be true yet depending on the feature you’re working on it’s not only possible but both cleaner and elegant. The catch here is that it’s usually only possible to do this for features that are designed for extensibility and have public API’s. The feature I’m currently working on is such a feature so we took advantage of that fact when writing tests for it. Unfortunately this feature is not public yet so I can’t talk about it right now but I’ll cover it in detail in a later post.

A brief overview of how the feature works. There’s a main object that implements an interface. This object in turn request information out of different objects that in turn implement a different interface and then does some operations with the results before displaying them to the user. So the code looks something like this:

Interface IMainObject

{

Initialize(object);

List<FeatureResults> GetInformation(String[]):

}

MainObject : IMainObject

{

private List<ISubObject> objects;

public Initialize(object);

public List<FeatureResults> GetInformation(String[] parameters):

};

The feature initializes an instance of MainObject and based on the user input the information is displayed on the UI. Now since these interfaces are designed for extensibility and made public we can use them to do our testing instead of asking the feature developers to add in some test hooks for us.

What I did for this feature was write a new library that implements these interfaces and plugged it in to our existing testing framework (which takes care of all the other things we need to actually run an automated test). One thing you’re probably wondering is how does the interface communicates with your application (Visual Studio in this case). That is taken care of by the initialize method of the IMainObject interface. In this case the parameter is not actually an object but one of the service providers that Visual Studio uses. And the next obvious question, how do we get that service provider? In order to get that we’re using one of our test libraries which is a Visual Studio package so it executes in proc and has a method that returns service providers.

As a recap, to use this method of test hooks your feature needs two things:

1)      A feature that was designed with extensibility in mind and has public API’s or interfaces.

2)      Public API’s from your application designed to allow external sources to write addons or packages for your application (so that you can communicate with the API’s from the feature).

Of course the drawback to this is that any changes done to the interfaces are going to break the test libraries. That being said the two major advantages are having almost full control of your test hooks and more importantly taking the testing part out of the feature owners’ hands. By doing so you’re making much better use of both sides’ resources as the feature owners can concentrate their efforts on the features and the testing side is not blocked by waiting for test hooks to be available.

Thanks,

Raul Perez

Visual C++ Team