In traditional unit test suites, each test case represents an exemplary usage scenario, and the assertions embody the relationship between the input and output.  Verifying a few such scenarios might well be enough, but experienced developers know that bugs lurk even in well-tested code, when correct but untested inputs provoke wrong responses.

Generating traditional unit tests

IntelliTest generates traditional unit test suites. When run on a piece of code-under-test, the IntelliTest engine tries to generate a test suite with high code coverage (in particular, it focuses on branch coverage by default). It does so by iteratively synthesizing precise concrete inputs that let it execute deeper and deeper into the code-under-test. The generated traditional test suite contains an individual test case for each such concrete input and assertions on the corresponding observed output. The inputs include edge cases that the developer might have missed.

*’How is that testing whether the code is correct or not? How useful is a test suite that provides high code coverage for an add method that might actually be doing a multiplication?’ *one might think.

Even without knowing about the intended and correct behaviour, the IntelliTest generated test suite describes the actual observed behaviour of the code under test. By itself this is useful as characterization tests, protecting existing behaviour against unintended changes.

Testing for correctness

Testing for correctness can be introduced naturally into this approach using assertions. Assertion statements (for e.g. Debug.Assert) are all compiled down to branches – an *if *statement with a *then *branch and an *else *branch representing the outcome of the predicate being asserted. Since the engine computes inputs to exercise branches, it ends up exercising such assertions as well. Thus, if the code-under-test contains assertions representing its correct behaviour, then IntelliTest ends up generating a test suite that validates such correctness. ***Assertions ***connect ***code coverage ***and ***correctness***.

One Test to rule them all

As a developer our understanding of the code-under-test will eventually grow beyond what can be represented by a few individual example test cases. By understanding the code-under-test and the individual unit tests, we might arrive at a more general representation of its behaviour. Instead of saying ‘for this input, this is the expected result’ we might be able to say ‘this method performs this kind of a computation on its inputs’ – instead of saying ‘codeUnderTest(2) should return 4’ one might be able to say ‘codeUnderTest takes an integer and doubles it’. We have hoisted exemplary usage scenarios to a level where they capture a general relationship between inputs and outputs. Such a relationship can be encapsulated in a method that looks like a test method but is universally quantified – any assertions we make must hold for all possible input values – universally quantified assertions or “for all” assertions, if you will. For example, the following is one such universally quantified method that asserts that after adding an element to a non-null list, the element is indeed contained in the list:

Assumptions work well in such cases, placing restrictions on the input data and exposing developer intent along the way, and IntelliTest provides a rich Assumptions API. The example shows the use of the special assertion API that IntelliTest provides, but we can as well use the ones that come with the test framework. IntelliTest emits a stub for such a universally quantified method – this is the parameterized unit test method (PUT or “the” IntelliTest) we discussed in the previous post. It serves as the partial specification for the code-under-test. Elaborating it does not require or introduce any new language or artifact. It’s written at the level of the actual APIs implemented by the software product, and in the programming language of the software product. IntelliTest exploration can be launched directly on the code-under-test or on this parameterized unit test to generate and update the suite of traditional unit tests. This unlocks the full potential of IntelliTest.

We started this post saying that IntelliTest tries to generate a suite with high code coverage. The organization and complexity of the code-under-test present IntelliTest several barriers to this, and in a future post we will show how they may be overcome.