Unit Testing Azure DevOps UI Extensions

Background

One of our automotive customers uses Azure DevOps for embedded software development. Process wise they need to comply with the A-SPICE standard (Automotive SPICE®) which regulates the software creation process and allows assessments and traceability for error root cause research in case of, for example, a car accident. Basically, it is a V-Model process where all upper and lower levels are linked with each other.

An example: a legal requirement is specified and linked to a technical requirement which specifies how to fulfill the legal constraints. The technical requirement could specify valid test result ranges of a computation. Since this all needs to be recorded, the computation (e.g. a program code file) needs to be linked to the technical requirement too. If the result of that program complies to the technical requirements the link is called valid or consistent.

Creating the links and checking the validity is a labor-intensive manual task. It is also not easy to track manually if a validation must be redone because the program code was changed. Therefore, the overall goal is to automate or at least assist manual validation as much as possible.

Azure Boards provides the possibility to highly customize software development processes and already includes linking features between any type of so-called Work Items (in this case a legal or technical requirement, epic, user stories etc.). Therefore, it was chosen as the foundation for the implementation.

In a first phase we helped develop the customization and logic to invalidate links to code artifacts if they were changed in a git-based code repository. This helps to identify which validations must be redone.

Errors in the automation tools could break the traceability aspect of A-SPICE and hence ultimately could result in legal consequences for our customer (e.g., when a required root cause examination could not be answered in court). Therefore we needed to achieve complete unit test code coverage for all parts of the automation tools. This will help identifying functional errors in early development, especially as these tools grow in complexity in the next phases.

Part of these automation tools is an Azure Boards Work Item UI extension based on the Typescript React Framework for Azure DevOps. In this post we will only focus on this Work Item UI extension and how to achieve unit testing with complete code coverage for it.

Requirements and Challenges

This post will demonstrate how we achieved the non-functional requirement for the development and what we did to achieve the functional requirement.

Functional Requirements:

Assist in finding linked code artifacts (stored in git repository) which need to be re-validated (VersionedItems)
Any requirement can have multiple owners

Non-Functional Requirement:

Achieve complete code coverage for all components of the developed automation tools

Challenges to achieve the Functional Requirement:

Azure Boards does not support git repository Versioned-Item links, therefore we had to develop a custom UI control
Azure Boards does not support multiple owners of a Work Item, therefore we had to develop a custom UI control

Both controls are packed together into one Azure DevOps Extension.

Challenges to achieve the Non-Functional Requirements:

Creating unit tests for Azure DevOps React/Typescript based UI components with complete code coverage

To unit test the components we need to cut off all external service dependencies. If an SDK method is used which leads to an API call there is a need to short cut the request and instead return a local dummy value for testing. To achieve the complete code coverage for the UI components there were following six sub-challenges to resolve these external service dependencies:

Mocking return values of Azure DevOps Extension SDK methods
- Azure DevOps environment related input parameters like current User, input parameter for the controls. How can we mock these parameters?

Mocking Azure DevOps Extension SDK service clients
- The SDK provides accessors to service clients. How can we mock these clients and their method results?

Triggering Azure DevOps Extension WorkItemFormUI event callbacks
- The controls implement callbacks (event handlers) to be informed about changes to the WorkItem (e.g. onSave, onDelete, onRefresh). These are triggered when the user is interacting with the WorkItemFormUI. How can we trigger them in the unit test?

Mocking return values and service clients of the Azure DevOps Extension API
- A service client for accessing the Git repos is used, how can we mock this client, and its methods return values?

Mock the external REST API calls and spy on the values transmitted
- Additional services were developed to achieve the functional requirement. How can we mock the generic REST Client we are using?

Mocking and spy on logging methods (Application Insights SDK)
- How can we verify that logs are correctly processed by the logging system?

Solution

Functional Requirements

We will not go into implementation details for the functional requirements in this blog post. The example Azure DevOps Extension (provided on GitHub) contains two Work Item UI controls:

Versioned Items Table — can link versioned items (code artifacts) from a distinct Azure Git repo branch to the work item. To do so it will save the state via a REST API external to Azure DevOps Services.
Multi Identity Picker — selects multiple identities from the Azure DevOps Organization associate AAD Tenant and saves the information into a text field of the work item.
Conceptual overview of the UI controls and their service interactions

The main technical difference between both components is that one will save the state externally and the other internally to Azure DevOps Services.

Both will use the TypeScript based Azure DevOps Extension SDK and API to communicate with Azure DevOps Service and do logging towards an Azure based Application Insights.

Non-Functional Requirement

This is the main technical content we focus on in this post. Step by step we show the code to test, the mocks needed and how the associate unit test looks like.

Mocking return values of Azure DevOps Extension SDK methods

The code: The Azure DevOps SDK must be initialized, after that we retrieve configuration values from it. We need to be able to specify the return values for the test (here: FieldName).

The mock: Due to the reactive style the SDK.init() function needs to return a resolved promise in order to process the lambda function in the .then() branch of the code. If we mock SDK.init() the configuration values would not be read at all. The getConfiguration() function can be easily mocked by returning a JSON object with the exact structure and data we expect. We added all configuration values for both UI components at once. If you additionally need to modify specific values within a test, it is possible to create an additional function mock which will be accessible in the unit test to overwrite the return value. In this example we do that for the RepositoryId parameter with a default value of ‘gitrepo’, by introducing a mocked function mockRepositoryId(). However, we are never changing the default value in the example tests. The information of the current user will be mocked with fixed values.

The test: The control will call componentDidMount() within the rendering process and will retrieve the parameter values set in the mock file. We don’t provide any assigned user data in this test. The default behavior is to use the current user as a value what is checked here.

Mocking Azure DevOps Extension SDK services

The code: We retrieve the Work Item Form Service with the function getService(). With this service we can access and modify the current work item opened in the UI. We want to retrieve the value of the field specified to hold the state of this component. This is done by the method getFieldValue().

The mock: We need two different services returned from getService() depending on the input parameters. In case the WorkItemFormService is requested we just return the getFieldValue() method as a mocked function mockGetFieldValue(). This allows us to inject the return value at the unit test level. In this case we do not distinguish what field is requested. If you are accessing several fields you would need to create an implementation for mockGetFieldValue() which will react to different input parameters.

As an example of how to react to input parameters you can have a look at the mocked searchIdentitiesAsync() method of the IdentityService. Depending on the query value we return another identity.

The test: Defines the return value for getFieldValue() and expects that the name associated to the email address will be displayed by the control. The mapping between email and display name happens in the already mocked searchIdentitesAsync() method.

Trigger Azure DevOps Extension WorkItemFormUI events

The code: When you develop a Work Item UI Control you can hook into events from the Work Item UI like saving, changed work item state etc. The control will react on the onSaved() event and register it’s callback to the SDK.

The mock: For this mock we had to figure out the exact type of the return value expected by the SDKs register() function. As you can see in the code snippet the return value will be assigned to the spyWorkItemCallBack function pointer from within the mocked register() function.

The test: Now we can trigger the callback function registered by the control within our tests by calling it via the spyWorkItemCallBack function pointer.

This will work in the same way for all other 5 event hooks possible to subscribe on.

Mocking return values and classes of the Azure DevOps Extension API

The code: One of the controls is loading all items of an Azure Git repo by utilizing the Azure DevOps Extension APIs Git Client. The control first has to load the Git Client with getClient() and only uses the getItems() method of this client.

The mock: The mock for the Git Client only contains the getItems() method and will return the value of the mockGetItems function if the RepositoryId is known, otherwise it will throw an error.

In order to retrieve the mocked Git Client the getClient() function from the Extension API has to be mocked as well. Otherwise, a call to getClient() will try to return an object of the original un-mocked class. If you use an additional API you will have to include it in this list.

The test: In the unit test it is now possible to define the return value of the Git Client getItems() method.

Mock the external Rest API calls and spy on the values transmitted

The code: In order to access external APIs we inherited the RestClientBase class of the Extensions API. We are mainly leveraging the _issueRequest() method provided by it and it’s data structures for issuing HTTP requests.

The mock: We mocked the base class in favor of our own client to really test our REST client’s code.

The mocked base class can now throw an error on demand, returning values we set at test level for specific API calls (evaluated by method and endpointURL) and even log the request values such as body and headers for a test validation in case of a POST call.

The test: We now can verify in a test a successful POST and if the data is transmitted correctly.

Mocking and spy on logging methods (Application Insights SDK)

The code: We log the exceptions and other information with a logger implementation of Application Insights. Here we log any exception occurring while communicating with the Git Rest Client.

The mock: We mock the method trackException() from the ApplicationInsights package and pass the exception as a parameter to the mockTrackException function.

In this way we can verify the logged exception content and can have complete code coverage for our logger implementation too.

The test: We can trigger an exception for the mocked Git getItems() method and validate that the exception was logged and the logged content is correct.

Summary

It is possible to mock all relevant parts of the Azure DevOps Extension SDK, Extension API as well as other external APIs to successfully create meaningful unit tests without external dependencies and reach full code coverage. Therefore, we could comply to the non-functional requirements and build out the foundation to identify errors in the A-SPICE automation tools logic early stage.

Additional resources:

The Team

Below is the Microsoft team that worked with me on this project, making sure we meet the customer requirements while coming up with a solution that works broadly all along the way!

Unit Testing Azure DevOps UI Extensions

Background