Toward the end of any customer engagement, the focus naturally shifts from feature development to validation and remediation of any discovered defects. While the software engineers who developed the features may feel confident that we’ve met the requirements, how do we prove this to stakeholders who are not familiar with the code base? Even with automated unit and integration tests it can be difficult to set up and walk through every key end-to-end user workflow–especially when there are many outside dependencies or long-running asynchronous processes.

Our team has developed an approach to acceptance tests for services that expose a set of HTTP APIs. This approach:

• Is easy to set up and use
• Helps to build an understanding of how to use the APIs by guiding the customer through example scenarios
• Leaves the customer with a reusable suite of tests
• Is not dependent on integration with a user interface

Implementation of Our Approach

While it’s common for developers to use Postman as a tool during API development, we typically stop at creating a basic set of requests that target each of the endpoints. Our team leverages Postman’s Collection feature to implement sets of requests that guide the user through specific end-to-end feature workflows. We also use Postman’s built-in test framework to make assertions on responses and to automatically save off values that will be needed in the subsequent requests. In addition to Postman Collections, we write documentation for each acceptance test so that the user has insight into the system’s expected behavior. We store both the documentation and the Postman Collection files in the same repository as the project’s codebase in order to make them accessible to anyone who may need them in the future. For each acceptance test, we follow the familiar Arrange/Act/Assert test pattern.

Acceptance Test Documentation

At the root of the repository’s Acceptance Test folder, we include a README that explains how to set up Postman for use with these tests. Each document is implemented as a markdown file that contains information and links, including internal references to project design records, that are relevant to that specific test.

The general format of an Acceptance Test Document includes:

• The test objective
  • A description of the use case, feature, or workflow
  • Links to backing documents, if applicable
• Preconditions and Troubleshooting
  • Links to set-up instructions in the README
  • Solutions to common issues
• The Postman Collection file for this test
  • The JSON that defines the set of requests when imported into Postman
• Instructions on how to run the test
  • Step-by-step descriptions of each request in the order they should be executed
  • An explanation of the expected response for each request
• Instructions for how to clean up the environment at the end of the test
  • Manual steps or DELETE requests sent via Postman depending on what’s appropriate and feasible for the project
  • This is important to support repeatability for the acceptance tests

Example Acceptance Test Document

Postman Environment

We first create a Postman Environment, which manages a set of variables that will be used across all acceptance test Collections. For example, a Postman Environment file should hold a value for the base URL used for requests, which can be set to point to instances of the hosted service in different environments (e.g. local, development, and test). The use of Postman Environment variables allows requests to be written in a manner that makes them reusable across these hosted environments. A basic Postman Environment file is exported as JSON and committed to the repository. A user can import and save multiple copies of a Postman Environment. This makes it easy to toggle between target environments using the drop-down selector in the Postman application.

Example Postman Environment

Postman Collection

When we create a new Collection in Postman, it is simple to export it as a JSON file. We then commit this file to the project repository’s Acceptance Test folder.

Each Postman Collection:

• Contains all the requests needed to walk through a specific user workflow or feature
• Organizes requests in the order they should be used
• Internally manages any variables needed during the acceptance test

A Postman Collection’s variables can be viewed or updated by navigating to the Variables pane.

Example Acceptance Test Postman Collection

Each Postman request has a Pre-request Script and Test section. The Pre-request Script is useful for retrieving values from the Collection variables or otherwise setting up preconditions for the execution of the request. In the example below, values for iterator and jobIdentifier are retrieved from the Collection variables and a value for jobId is saved. Notice that the URL section of the request in the example uses variables from the Postman Environment as well as variables from the Postman Collection.

Example Postman Pre-request Script

In the Test pane of a request, the built-in Postman test framework allows you to write JavaScript to interact with and make assertions on the response to the request. Each test shows a green PASS or red FAIL icon, which makes it easy for a user to determine the outcome. In the example below we see a request that “polls” the status of a long-running asynchronous process. When the process is not yet complete, two of the tests fail. Once the process is complete, all tests pass.

Example Postman Test Pane

Conclusion

The end-to-end workflows and assertions in the acceptance test Postman Collections help software engineers on the project to validate what we’ve built. In some cases the acceptance tests reveal software defects that are difficult or impossible to detect with traditional unit tests. If we uncover defects, we can address them and re-run any failing Acceptance Tests to confirm that the software now works as expected. Our acceptance test cases provide additional test coverage, which improves our confidence in the final product.

We’ve also received positive feedback from our customers. They are generally familiar with Postman, which is free and easy to use. The step-by-step instructions we provide in the acceptance test documentation and the ready-to-import Postman Collection files make it easier for customers to validate the covered use cases and confirm that the requirements have been met. Because the documentation and Postman Collection files are stored in the project repository, the customer can re-use, expand, or update them as they continue to develop the project.

References

• Postman Environments
• Postman Collections
• Postman Pre-request Script
• Postman Test Scripts

Acknowledgements

I want to give thanks to crew members who worked to implement and refine our acceptance test approach: Bryan Leighton, Kanishk Tantia, Cameron Taylor, Uffaz Nathaniel, and Avin Zarlez.