Inside the making of the Azure SDK management libraries

Elendil Zheng

As a cloud native developer, which types of developer experiences do you care about the most? Interacting with cloud resources in your app is certainly at the top of your experiences list—whether sending a message to an Azure Service Bus endpoint or creating new VM instances to carry heavier workloads. You can use a cloud SDK, like the Azure SDK, for all these cloud service interactions, but how is the Azure SDK generated? What’s the difference between a cloud SDK and a traditional SDK? Let’s start our journey!

Differences between traditional SDK and cloud SDK

A traditional SDK, like the .NET SDK or Java SDK, are program interfaces to the runtime of .NET (CLR) or Java (JVM). The communication protocol is within a process level, usually in binary format.

A cloud SDK usually starts with an OpenAPI definition. OpenAPI is more like a cross-process communication specification that mainly relies on RESTful HTTP APIs. OpenAPI is everywhere in the world of cloud, but OpenAPI has its own disadvantages.

Problems with using OpenAPI to interact with Azure

OpenAPI is an interface definition to a REST API service. It isn’t an out-of-the-box solution for any one programming language. When developers start using OpenAPI, they’ll encounter the following challenges:

• Authentication to be able to use the REST API
• No native IDE IntelliSense to clarify the required parameters for operations in the REST API
• Developers need to handle every detail of calling the REST API, such as:
  • Exception handling
  • Long-running operation handling
  • Paging operation handling
  • HTTP client lifecycle management

Writing solid code to support the above functions could be a tough task. Therefore, using an SDK becomes a natural choice.

How we generate the Azure SDK from OpenAPI

The Azure SDKs are designed to conform to the General SDK library guidelines, which strive to make the SDK development experience consistent across Azure services. We have started a phased deprecation of Azure SDK libraries that don’t conform to the latest library design guidelines. To see which packages have been deprecated, see the deprecated release tracking page.

To make an SDK for a new Azure service, the Azure SDK team works closely with the service team. The typical steps are:

1. The Azure service team defines the OpenAPI specification (Swagger file) and creates a PR in the REST API specification repository.
2. We review and validate the OpenAPI spec and check if there are any breaking changes, syntax errors, etc.
3. The AutoRest tool is used to generate a client library from the OpenAPI specification. AutoRest is an open-source tool created by Microsoft that generates client libraries for accessing RESTful web services. Language-specific extension packages can be added on AutoRest to generate a client library for a specific language.
4. We review the generated SDK and raise a PR to the SDK’s repo. We have a solid engineering system that runs tests against these PRs to make sure we don’t bring any regression issues forward. The test cases include handwritten tests, but in the future, we’ll have more auto-generated test cases.
5. Once all the checks pass in the PR pipeline, the generated client library can be merged into the specific language SDK repo.
6. After the client library is merged into the language SDK repo, the CI/CD pipeline is triggered. If pipeline execution is successful, the new client library is released to the language’s package manager.
7. All SDK-related docs on docs.microsoft.com are updated with the latest client library sample code.

AutoRest plays a key role in the client library generation for new Azure services. It knows how to combine common patterns of REST API calls and exposes these as more developer-friendly methods in the client library. For example, long-running operations and some resource creation and deletion requests can’t be carried out immediately. In such a situation, the server sends a 201 (Created) or 202 (Accepted) and provides a link to monitor the status of the request. This approach makes sense when we’re directly calling a RESTful API. From a programming language point of view, it’s problematic. Instead, this kind of long-running RESTful API call needs to be converted into an async function with a callback. The callback allows the developer to handle the function execution success or exception. OpenAPI, unfortunately, doesn’t have an equivalent built-in async definition. To fill this gap, Microsoft defined a set of AutoRest OpenAPI extensions to enrich the OpenAPI semantics.

Now, let’s get back to our long-running operation scenario. We can specify an x-ms-long-running-operation header in our OpenAPI definition for a specific operation. For example, below is part of the OpenAPI definition for Azure VM creation/update. The full definition can be found here.

"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{vmName}": {
      "put": {
        "tags": [
          "VirtualMachines"
        ],
        "operationId": "VirtualMachines_CreateOrUpdate",
        "description": "The operation to create or update a virtual machine. Please note some properties can be set only during virtual machine creation.",
        "parameters": [
          {
            "name": "resourceGroupName",
            "in": "path",
            "required": true,
            "type": "string",
            "description": "The name of the resource group."
          },
          {
            "name": "vmName",
            "in": "path",
            "required": true,
            "type": "string",
            "description": "The name of the virtual machine."
          },
          {
            "name": "parameters",
            "in": "body",
            "required": true,
            "schema": {
              "$ref": "#/definitions/VirtualMachine"
            },
            "description": "Parameters supplied to the Create Virtual Machine operation."
          },
          {
            "$ref": "#/parameters/ApiVersionParameter"
          },
          {
            "$ref": "#/parameters/SubscriptionIdParameter"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "$ref": "#/definitions/VirtualMachine"
            }
          },
          "201": {
            "description": "Created",
            "schema": {
              "$ref": "#/definitions/VirtualMachine"
            }
          },
          "default": {
            "description": "Error response describing why the operation failed.",
            "schema": {
              "$ref": "#/definitions/CloudError"
            }
          }
        },
        "x-ms-long-running-operation": true
      }
    }

With the OpenAPI definition above, here’s the generated client function call to BeginCreateOrUpdate in Golang. The full sample code can be found here.

pollerResponse, err := vmClient.BeginCreateOrUpdate(ctx, resourceGroupName, vmName, parameters, nil)
    if err != nil {
        return nil, err
    }

    resp, err := pollerResponse.PollUntilDone(ctx, 10*time.Second)
    if err != nil {
        return nil, err
    }
}

The poller class in the preceding sample comes from Azure Core. Azure Core provides the infrastructure of the whole Azure SDK, such as HTTP client lifecycle management and exception handling. With the power of Azure Core, we can apply more OpenAPI extensions in our OpenAPI specification and turn that into a more developer-friendly SDK. Such as:

• x-ms-error-response: Indicates whether the response status code should be treated as an error response
• x-ms-pageable: allows paging through lists of data

For more extension flags, see AutoRest Extensions for OpenAPI 2.0.

Conclusion

We discussed the benefits of using a cloud SDK and described our process for making an SDK for Azure developers. Thank you for reading this blog post.

Elendil Zheng Program Manager II, Azure SDK

Follow