When a library has dependencies with semantic version or semver
ranges, how do you ensure and validate that your library actually supports the version ranges you claim? For instance, if we have @azure/keyvault-keys that supports 1.0.0 to 1.0.4 version ranges of @azure/core-http, how would you ensure that this library is actually compatible with all versions of core-http from 1.0.0 to 1.0.4 ( 1.0.0 , 1.0.1, 1.0.2, 1.0.3 and 1.0.4)? One solution is to test the library against each version of the dependency that you claim to support. But as we keep releasing newer versions, the list of versions that we want to test will keep increasing. Given this, the testing of the entire version range for all dependencies may not always be feasible and we want to find an optimum scalable solution for the dependency testing in this scenario. One approach to this is “min-max” testing, or testing the library against the minimum version of dependency version range and against the maximum version of the dependency version range.
We would like to share with you the journey of the Azure SDK team with the min-max dependency testing for JavaScript SDKs. By doing the min-max testing, we are making sure that the acceptable dependency version ranges are always updated. This ensures that the users aren’t inadvertently broken when there is a mistake on the acceptable version ranges, thus giving the Azure SDK team the confidence to iterate rapidly on new features and fixes based on user feedback.
Approaches to Dependency Testing
On the Azure SDK Team, the ideal way to test the compatibility of libraries with different versions of their dependencies would be to create a test application emulating client scenarios. After brainstorming several ideas of a “client application” we shortlisted the options: we could use the existing smoke tests which already behave similarly to a typical application, or we could restructure the existing functional test assets to work for our purposes. We test only a few functions with the smoke tests, which means we may miss out on some scenarios for the minimum/maximum dependency testing. Hence doing the former option, meant getting very limited test coverage as compared to our already existing test coverage.
Taking the second option seemed the ideal approach, but it had its own set of challenges. In the JavaScript world, tests directly reference the source files. In the internal source code, we have internal TypeScript type declarations which are not exposed and the tests directly reference those. The major challenge for us was to make the min-max testing work with the existing test and build setup and doing minimal changes to the test collateral and other infrastructure.
Finding an Ideal Solution
The current structure of our Azure SDK for JS/TS package(eg : @azure/keyvault-keys folder structure) is as follows :
- samples
- review
- src
- test
- testfile 1
- testfile 2
- testfile n
- package.json
- rollup config files
- readme.md
- license.txt
To run tests that emulate a typical application, we need to construct a test package that has its own package.json with dependencies on various versions of our packages. In addition, we need to figure out which tests from the test suite can be run for min-max testing and find a way to build and run the test package that is compatible with our existing build process.
Using Package References
In order to make the Azure SDK library a dependency of the test package, we need to ensure that the test files are not directly referencing the source code but they are using package references and ensure if it’s actually referencing the code from the Azure SDK library installed from NPM into the local node-modules folder. Most of our tests are either testing the internal source code or are testing the public APIs. The internal tests will always need to reference the source code, thus for the purpose of min-max testing, we only need to run the tests for public APIs. Given the structure of the SDK test folder we didn’t have a clear separation between the files doing internal v/s public APIs testing, which now became necessary.
Separating Tests into Internal and Public
To distinguish between internal and public tests, just changing source references to package references was not enough. Some test files need to be rewritten in a decoupled fashion so that the tests for public/exported APIs are separated from the tests for internal code and test utilities. This separation of tests became a challenge since all of our existing test code now needed a restructuring and most of the changes could not be automated. We then organized the tests for all the Azure Key Vault libraries into the following structure :
- test
- internal ( tests for internal references)
- public (tests for public APIs)
- utils (utility functions)
- readme.md
After the test separation, we could easily replace the source references with references to package (eg : @azure/keyvault-keys) in the test files to run the tests.
Converting Tests Folder into Package
Azure SDK team’s main goal is to make a test package behave similar to how a client application would behave on a user’s machine. The creation of a custom package.json for public APIs test folder is an implementation detail of this goal. The package.json would take dependency on the package itself (eg: @azure/keyvault-keys) and all of its depedencies and dev-dependencies. We could either manually create and commit the package.json or autogenerate it within the pipeline for doing min-max testing.
Committing the package.json was interfering with the internal developer toolchain and workflow, hence generating package.json by automation within the pipeline seemed to be the way out. Taking the example of @azure/keyvault-keys, the package.json has the following dependencies:
{
"name": "@azure/keyvault-keys",
"sdk-type": "client",
"author": "Microsoft Corporation",
"version": "4.1.0",
"license": "MIT",
"dependencies": {
"@azure/core-http": "^1.1.6",
"@azure/core-lro": "^1.0.2",
"@azure/core-paging": "^1.1.1",
"@azure/core-tracing": "1.0.0-preview.9",
"@azure/logger": "^1.0.0",
"@opentelemetry/api": "^0.10.2",
"tslib": "^2.0.0"
}
}
When doing the minimum dependency testing we create a package.json in the public test folder and take the minimum matching semantic version (or semver
) for the Azure SDK depedencies or dev-dependencies. We let the other dependencies and dev-dependencies be as is and also add a dependency on the current version of the @azure/keyvault-keys that we want to test. So the new package.json would look like –
{
"name": "azure-keyvault-keys-test",
"sdk-type": "client",
"version": "0.1.0",
"dependencies": {
"@azure/keyvault-keys": "4.1.0",
"@azure/core-http": "1.1.6",
"@azure/core-lro": "1.0.2",
"@azure/core-paging": "1.1.1",
"@azure/core-tracing": "1.0.0-preview.9",
"@azure/logger": "1.0.0",
"@opentelemetry/api": "^0.10.2",
"tslib": "^2.0.0"
}
We do similarly for maximum dependency testing, except we we take the maximum matching semver (that is published on NPM) for each Azure SDK dependencies or dev-dependencies.
Building the Test Package and Running the Tests
For the Azure SDK for JavaScript and TypeScript libraries, we use Rush for build orchestration and dependency management and alignment across all the SDKs within the mono-repo. Rush gives us the ability to ensure that all libraries that are part of the Rush workspace use the same version ranges for their shared dependencies. Rush also gives us the ability to take dependency on another library in the same workspace by just specifying it’s version in the dependency list, even when it’s not published on NPM. To build and run the test package we had to ensure that the test package is NOT directly referencing the internal bits of the source code of the SDK package, but rather importing it. Since we are trying to emulate a client package, majority of the users use npm. Usually we want to test using the same tools that our users use. So we had choices to either use npm commands
or Rush
for the build process.
Building the test package using the npm
command caused issues as we used Rush for the mono-repo and the SDK package’s node-modules folder contained only symbolic links to the dependencies into the central pnpm location instead of actual dependencies inside the node-modules folder. With the test folder being nested inside the SDK package folder, the presence of an outer node-modules folder was adding complications to the path of the dependencies for test-package. Due to this structure, the transitive dependencies would not always be included in the node-modules when we had to take a dependency on an @azure scoped package within the same workspace, not published to NPM. On the other hand, adding the test package to the Rush suite wasn’t as terrible as we initially thought. Rush does enforce all the packages in the same workspace to have the same dependency versions. But in this case, we wanted the test package to explicitly use the versions of dependencies we wanted to test. We could specify these dependency versions in the allowedAlternativeVersions
section of the common-versions.json config file, thus enlisting them as exceptions.
Automation for the Process
To run the min-max dependency tests, we automated the entire process in our CI pipeline, except for the separation of tests. The final process looks like:
- Insert a package.json for public test folder using the approach described above
- Add tsconfig for public test folder with the same compiler options as the original package’s tsconfig.
- Replace source references with references to package in the import statements of the test files of the public test folder
- Update rush config file rush.json to add the test package to the rush suite
- Add the minimum matching semver (or maximum) dependency versions as exceptions under the
allowedAlternativeVersions
section of the common-versions.json config file for rush
The test package now behaves like a client application and takes dependency on minimum and maximum matching semver of the dependencies and we can run build and run tests using rush commands. If there is an incompatibility with the dependency version we are testing, the tests for public APIs will fail, else they will succeed. We run these min-max tests within our nightly test pipelines. We can then update the dependency version ranges that are supported with the SDK which ensures the users have an acurrate up-to-date version lists of dependencies they can use. Thus the Azure SDK team can iterate rapidly on new features and fixes for the customers.
Azure SDK Blog Contributions
Thank you for reading this Azure SDK blog post! We hope that you learned something new and welcome you to share this post. We are open to Azure SDK blog contributions. Please contact us at azsdkblog@microsoft.com with your topic and we’ll get you setup as a guest blogger.
Azure SDK Links
- Azure SDK Website: aka.ms/azsdk
- Azure SDK Intro (3 minute video): aka.ms/azsdk/intro
- Azure SDK Intro Deck (PowerPoint deck): aka.ms/azsdk/intro/deck
- Azure SDK Releases: aka.ms/azsdk/releases
- Azure SDK Blog: aka.ms/azsdk/blog
- Azure SDK Twitter: twitter.com/AzureSDK
- Azure SDK Design Guidelines: aka.ms/azsdk/guide
- Azure SDKs & Tools: azure.microsoft.com/downloads
- Azure SDK Central Repository: github.com/azure/azure-sdk
- Azure SDK for .NET: github.com/azure/azure-sdk-for-net
- Azure SDK for Java: github.com/azure/azure-sdk-for-java
- Azure SDK for Python: github.com/azure/azure-sdk-for-python
- Azure SDK for JavaScript/TypeScript: github.com/azure/azure-sdk-for-js
- Azure SDK for Android: github.com/Azure/azure-sdk-for-android
- Azure SDK for iOS: github.com/Azure/azure-sdk-for-ios
- Azure SDK for Go: github.com/Azure/azure-sdk-for-go
- Azure SDK for C: github.com/Azure/azure-sdk-for-c
- Azure SDK for C++: github.com/Azure/azure-sdk-for-cpp
0 comments