Portable Compression and HttpClient Working Together

Today we’re happy to announce that we released two NuGet packages:

A brand new Microsoft.Bcl.Compression (Beta). A portable compression library that supports stream based compression (gzip, deflate) as well as ZIP archives.
An updated Microsoft.Net.Http 2.2 (Beta). Last week we shipped an RTM version. Today, we released a new beta that provides automatic decompression by taking a dependency on the Microsoft.Bcl.Compression package.

Before we go into the details, let’s first take a look at why compression is particularly interesting for HttpClient.

Compression and HttpClient

We live in a world where we are permanently surrounded by devices, particularly smart phones. Apps that run on these devices are often not super useful in isolation; they require services to provide data and enhance their features.

As a result many developers use the HttpClient class to access web resources such as REST services. Most service providers want to minimize the data that is being transmitted between a client and server. This is particularly helpful when services are accessed by apps on mobile devices that use metered Internet connections. The cost to use the app is directly affected by the amount of data being transmitted, and less data means lower cost to the end-user.

In HTTP, headers provide a way to negotiate capabilities between the client and server. The Accept-Encoding header allows clients to tell the server that they support reading the response in a compressed format (e.g., gzip or Deflate), and is a logical step when using HTTP from a mobile client. The server may choose to ignore the header or reply in another format the client indicates support for.

Not surprisingly, after we announced a portable HttpClient, you asked us to support the AutomaticDecompression property on HttpClientHandler, particularly for Windows Phone apps. Unfortunately, at the time, we didn’t have a portable compression library at our disposal.

In this blog post, I’d like to show you our new portable compression library and explain how we leverage it to support AutomaticDecompression in HttpClient.

Enter Microsoft.Bcl.Compression

Today we released a beta of Microsoft.Bcl.Compression, a NuGet package that provides compression APIs in a portable class library. The compression support covers the following aspects:

Compression streams (Deflate & gzip)
Reading and writing ZIP archives

We also released an update to our HttpClient NuGet package that takes a dependency on Microsoft.Bcl.Compression to support automatic decompression.

The Microsoft.Bcl.Compression NuGet package is supported on the following platforms:

.NET Framework 4.5
Windows Phone 8
.NET for Windows Store apps

This package also includes portable class libraries that target any of those platforms.

The platform list is different from the platforms supported by HttpClient. In particular, HttpClient supports the .NET Framework 4 and Windows Phone 7.5, but the Microsoft.Bcl.Compression package isn’t supported on those platforms.

Using Microsoft.Bcl.Compression directly

Of course, a portable compression library has plenty of uses other than just reducing the footprint of networking requests. That’s why Microsoft.Bcl.Compression is completely independent from HttpClient and can be used directly. Let’s have a look at how you would use it.

Consider a Windows Phone app that shows a random poem from a collection of poems. The poems are represented as text files. In order to keep the app small, we compressed all poems using gzip. During application startup, a random poem is selected and read from the application data. In order to decompress the contents of the poem, we use the GZipStream class.

        private static string ReadRandomPoem()
        {
            int randomPoemNumber = 42; // Super random
            string poemName = string.Format("Poem{0}.txt.gz", randomPoemNumber);
            using (Stream stream = OpenApplicationFile(poemName))
            using (Stream decompressed = new GZipStream(stream, CompressionMode.Decompress))
            using (StreamReader reader = new StreamReader(decompressed))
            {
                string text = reader.ReadToEnd();
                return text;
            }
        }

Now let’s say that in the next version of the app we allow the user to select a form of poem, such as sonnet or haiku. To implement this, we decide to store all poems that share the same form in the same ZIP archive. To read a ZIP archive we use the ZipArchive class.

        private static string ReadRandomPoem()
        {
            int randomPoemNumber = 42; // Still super random
            string poemName = string.Format("Poem{0}.txt", randomPoemNumber);
            using (Stream archiveStream = OpenApplicationFile("haikus.zip"))
            using (ZipArchive archive = new ZipArchive(archiveStream, ZipArchiveMode.Read))
            {
                ZipArchiveEntry zipArchiveEntry = archive.GetEntry(poemName);
                using (Stream stream = zipArchiveEntry.Open())
                using (StreamReader reader = new StreamReader(stream))
                {
                    string text = reader.ReadToEnd();
                    return text;
                }
            }
        }

Using automatic decompression with HttpClient

Now let’s have look at how you would use the decompression APIs with HttpClient.

First, automatic decompression is not enabled by default for HttpClient. To use it, you need to set the AutomaticDecompression property on HttpClientHandler to GZip or Deflate. This API follows the optional feature pattern; meaning it exposes an API that indicates whether a particular feature is supported. Automatic decompression support is indicated with the SupportsAutomaticDecompression property.

If an implementation of HttpClient doesn’t support automatic decompression, it returns false from the SupportsAutomaticDecompression property and throws a NotSupportedException from the setter and getter for the AutomaticDecompression property. This was how our previous release of the HttpClient NuGet package behaved.

With this pattern in mind, using automatic decompression looks like this:

        var handler = new HttpClientHandler();
        if (handler.SupportsAutomaticDecompression)
        {
            handler.AutomaticDecompression = DecompressionMethods.GZip |
                                             DecompressionMethods.Deflate;
        }
        var httpClient = new HttpClient(handler);
        var str = await httpClient.GetStringAsync("http://en.wikipedia.org/wiki/Gzip");

The request then provides the additional Accept-Encoding header:

        GET http://en.wikipedia.org/wiki/Gzip HTTP/1.1
        Host: en.wikipedia.org
        Accept-Encoding: gzip, deflate
        Connection: Keep-Alive

Servers that choose to support it will respond and indicate the compression algorithm they used. In this example, the server compressed the body using gzip:

        HTTP/1.1 200 OK
        Server: nginx/1.1.19
        Date: Wed, 13 Mar 2013 14:04:24 GMT
        Content-Type: text/html; charset=UTF-8
        Content-Length: 17765
        Connection: keep-alive
        X-Content-Type-Options: nosniff
        Content-Language: en
        Last-Modified: Tue, 05 Mar 2013 03:38:51 GMT
        Content-Encoding: gzip
        Expires: Thu, 01 Jan 1970 00:00:00 GMT
        Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
        Vary: Accept-Encoding,Cookie
        Age: 179

The response stream that HttpClient returns to you will automatically decompress the result using the appropriate algorithm. On my machine I got the following results:

This yields to a reduction by 77%. Needless to say, the actual results depend on the request and how well it compresses. So your results will naturally vary.

The CPU architecture matters

In the .NET Framework 4.5 we changed our DeflateStream implementation to use the popular zlib library. As a result, DeflateStream provides a better implementation of the deflate algorithm and, in most cases, a better compression than in earlier versions of the .NET Framework.

Microsoft.Bcl.Compression also uses the deflate algorithm, which is implemented in native code. This means that Microsoft.Bcl.Compression requires CPU-specific binaries, and each project that consumes it has to be CPU specific. Given that some of the platforms the Microsoft.Bcl.Compression package supports already provide built-in support for compression, only Windows Phone projects are affected by this requirement.

However, we’ve found a way to avoid making each project CPU-specific:

Class library projects that consume Microsoft.Bcl.Compression can remain Any CPU. This makes them consumable by other Any CPU projects as well as CPU-specific projects.
Windows Phone application projects that use Microsoft.Bcl.Compression (either directly or indirectly via another class library) must add a NuGet reference to Microsoft.Bcl.Compression, and specify either ARM or x86 architecture, for phone or emulator respectively. This ensures we deploy binaries that match the CPU architecture of the application.

We understand that this can be tricky to get right. Therefore, our NuGet package will guide you through the process. Let’s assume that you have a solution with the following two projects:

Windows Phone App (configured as Any CPU)
Portable class library (configured as Any CPU)

Let’s say the portable class library depends on Microsoft.Bcl.Compression, whereas the phone app doesn’t depend on Microsoft.Bcl.Compression but does depend on the portable class library.

Building the solution will yield the following error message:

Phone Application: Project must install NuGet package Microsoft.Bcl.Compression.

After adding the reference to Microsoft.Bcl.Compression and rebuilding your app, a new MSBuild task (which comes with the NuGet package) will deploy CPU-specific binaries as dependencies of your app. If your app is configured as Any CPU, the build will fail again with this error message:

Phone Application: Microsoft.Bcl.Compression does not support the currently selected platform of ‘AnyCPU’. The supported platforms are ‘x86’ and ‘ARM’.

The project should build successfully after you change the platform target to X86 or ARM.

We’d love to hear your feedback around this!

Summary

We’ve released a beta version of the new Microsoft.Bcl.Compression package that provides portable compression support for your apps. When you’re using portable compression, don’t forget to set your CPU architecture for Windows Phone projects. And as you saw earlier: even if you forget it, our package will take care of reminding you.

We’ve also released a new beta for the existing Microsoft.Net.Http package in version 2.2. It adds support for AutomaticDecompression. Since this added substantial functionality to HttpClient, we decided to ship a beta instead of updating the stable release we shipped a week ago.

Here is our tentative release schedule for turning Microsoft.Net.Http 2.2 into a stable release:

End of June: RC of HttpClient 2.2 with automatic decompression
Around July (depending on feedback): RTM of HttpClient 2.2 with automatic decompression

This timeline depends on your feedback. So please let us know what you think of Microsoft.Bcl.Compression, and how the new version of HttpClient works for you!