November 21st, 2023

Announcing .NET Chiseled Containers

Richard Lander
Richard Lander
Product Manager

.NET chiseled Ubuntu container images are now GA and can be used in production, for .NET 6, 7, and 8. Canonical also announced the general availability of chiseled Ubuntu containers. Chiseled images are the result of a long-term partnership and design collaboration between Canonical and Microsoft. We announced chiseled containers just over a year ago, as a new direction. They are now ready for you to use in your production environment and to take advantage of the value they offer.

The images are available in our container repos with the following tag: 8.0-jammy-chiseled. .NET 6 and 7 variants differ only by version number. These images rely on Ubuntu 22.04 (Jammy Jellyfish), as referenced by jammy in the tag name.

We made a few videos on this topic over the last year, which provide a great overview:

We also published a container workshop for .NET Conf that uses chiseled containers for many of its examples. The workshop also uses OCI publish, which pairs well with chiseled containers.

Chiseled images

General-purpose container images are not the future of cloud apps

The premise of chiseled containers is that container images are the best deployment vehicle for cloud apps, but that typical images contain far too many components. Instead, we need to slice away all but the essential components. Chiseled container images do that. That helps — a lot — with size and security.

The number one complaint category we hear about container images is around CVE management. It’s hard to do well. We’ve built automation that rebuilds .NET images within hours of Alpine, Debian, and Ubuntu base image updates on Docker Hub. That means the images we ship are always fresh. However, most users don’t have that automation, and end up with stale images in their registries that fail CVE scans, often asking why our images are stale (when they are not). We know because they send us their image scan reports. There has to be a better way.

It’s really easy to demo the difference, using anchore/syft (using Docker).

Those commands show us the number of “Linux components” in three images we publish, for Debian, Ubuntu, and Alpine, respectively.

$ docker run --rm anchore/syft mcr.microsoft.com/dotnet/runtime:8.0 | grep deb | wc -l
92
$ docker run --rm anchore/syft mcr.microsoft.com/dotnet/runtime:8.0-jammy | grep deb | wc -l
105
$ docker run --rm anchore/syft mcr.microsoft.com/dotnet/runtime:8.0-alpine | grep apk | wc -l
17

One can guess that it’s pretty easy for a CVE to apply to one of these images, given the number of components. In fact, .NET doesn’t use most of those components! Alpine shines here.

Here’s the result for the same image, but chiseled.

$ docker run --rm anchore/syft mcr.microsoft.com/dotnet/runtime:8.0-jammy-chiseled | grep deb | wc -l
7

That gets us down to 7 components. In fact, the list is so short, we can just look at all of them.

$ docker run --rm anchore/syft mcr.microsoft.com/dotnet/runtime:8.0-jammy-chiseled | grep deb
base-files                                         12ubuntu4.4               deb     
ca-certificates                                    20230311ubuntu0.22.04.1   deb     
libc6                                              2.35-0ubuntu3.4           deb     
libgcc-s1                                          12.3.0-1ubuntu1~22.04     deb     
libssl3                                            3.0.2-0ubuntu1.12         deb     
libstdc++6                                         12.3.0-1ubuntu1~22.04     deb     
zlib1g                                             1:1.2.11.dfsg-2ubuntu9.2  deb

That’s a very limited set of quite common dependencies. For example, .NET uses OpenSSL, for everything crypto, including TLS connections. Some customers need FIPS compliance and are able to enable that since .NET relies on OpenSSL.

Native AOT apps are similar, but need one less component. We care so much about limiting size and component count that we created an image just for it, removing libstdc++6. This image is new and still in preview.

$ docker run --rm anchore/syft mcr.microsoft.com/dotnet/nightly/runtime-deps:8.0-jammy-chiseled-aot | grep deb | wc -l
6

As expected, that image only contains 6 components.

Chiseled images are also much smaller. We can see that, this time with (uncompressed) aspnet images.

$ docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}" | grep mcr.microsoft.com/dotnet/aspnet
mcr.microsoft.com/dotnet/aspnet               8.0-jammy-chiseled                      110MB
mcr.microsoft.com/dotnet/aspnet               8.0-alpine                              112MB
mcr.microsoft.com/dotnet/aspnet               8.0-jammy                               216MB
mcr.microsoft.com/dotnet/aspnet               8.0                                     217MB

In summary, Chiseled images:

  • Slice a little over 100MB (uncompressed) relative to existing Ubuntu images.
  • Match the size of Alpine, the existing fan favorite for size.
  • Are the smallest images we publish with glibc compatibility
  • Contain the fewest components, reducing CVE exposure.
  • Are a great choice for matching dev and prod, given the popularity of Ubuntu for dev machines.
  • Have the strongest support offering of any image variant we publish.

Distroless form factor

We’ve been publishing container images for nearly a decade. Throughout that time, we’ve heard regular requests to make images smaller, to remove components, and to improve security. Even as we’ve improved .NET container images, we’ve continued to hear those requests. The fundamental problem is that we cannot change the base images we pull from Docker Hub (beyond adding to them). We needed something revolutionary to change this dynamic.

Many users have pointed us to Google Distroless over the years.

“Distroless” images contain only your application and its runtime dependencies. They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution.

That’s a great description, taken from the distroless repo. Google deconstructs various Linux distros, and builds them back as atoms, but only using the most necessary atoms to run apps. That’s brilliant.

We have a strong philosophy of only taking artifacts from and aligning with the policies of upstream distros, specifically Alpine, Debian, and Ubuntu. That way, we have the same relationship with the upstream provider (like Ubuntu) as the user. If there is an issue (outside of .NET), we’re looking for resolution from the same singular party. That’s why we never adopted Google Distroless and have been waiting for something like Ubuntu chiseled, from the upstream provider.

Ubuntu chiseled is a great expression of the distroless form factor. It delivers on the same goals as the original Google project, but comes from the distro itself.

You might worry that this is all new and untested and that something is going to break. In fact, we’ve been delivering distroless images within Microsoft for a few years. At Microsoft, we use Mariner Linux. A few years ago, we asked the Mariner team to create a distroless solution for our shared users. Microsoft teams have been hosting apps in production using .NET + Mariner distroless images since then. That has worked out quite well.

Security posture

The two most critical components missing from these images are a shell and a package manager. Their absence really limits what an attacker can do. curl and wget are also missing from these images. One of the first things an attacker typically does is download a shell script (from a server they control) and then runs it. Yes, that really happens. That’s effectively impossible with these images. The required components to enable that are just not there and not acquirable.

We also ship these images as non-root.

$ docker inspect mcr.microsoft.com/dotnet/aspnet:8.0-jammy-chiseled | grep User
            "User": "",
            "User": "1654",

All new files and directories are created with a UID and GID of 0 — Dockerfile reference

This change further constrains the type of operations that are allowed in these images. On one hand, a non-root user isn’t able to run apt install commands, but since apt isn’t even present, that doesn’t matter. More practically, the non-root user isn’t able to update app files. The app files will be copied into the container as root, making it impossible for the non-root user to alter them or to add files to the same directory. The non-root user only has read and execute permissions for the app.

You might wonder why it took us so long to support these images after announcing them over a year ago. That’s a related and ironic story. Many image scanners rely on scanning the package manager database, but since the package manager was removed, the required database was removed too. Ooops! Instead, our friends at Canonical had to synthesize a package manager database file through other means, as opposed to bringing part of the package manager back just to enable scanning. In the end, we have an excellent solution that optimizes for security and for scanning, without needing to compromise either. All of the anchore/syft commands shown earlier in the post are evidence that the scanning solution works.

App size

There are multiple ways to control the size of container images. The following slide from our .NET Conf presentation demonstrates that with a sample app.

Image sizes with various options

There are two primary axis:

  • Base image, for framework-dependent apps.
  • Publish option, for self-contained apps.

On the left, the chiseled variants of aspnet result in significant size wins. The smaller chiseled image — “composite” — derives its additional reductions by building parts of the .NET runtime libraries in a more optimized way. That will be covered in more detail in a follow-up post.

On the right, self-contained + trimming drops the image size a lot more, since all the unused .NET libraries are removed.

Native AOT drops the image size to below 10MB. That’s a welcome and shocking surprise. Note that native AOT only works for console apps and services, not for web sites. That may change in a later release.

You might be wondering how to select between these choices.

  • Framework dependent deployment has the benefit of maximum layer sharing. That means sharing copies of .NET in your registry and within a single machine (if you host multiple .NET apps together). Build times are also shorter.
  • Self-contained apps win on size and registry pull, but have more limited sharing (only runtime-deps is shared).

Adoption

Chiseled images are the biggest change to our container image portfolio since we added support for Alpine, several years ago. We recommend that users take a deeper look at this change.

Note: The chiseled images we publish don’t include ICU or tzdata, just like Alpine (except for “extra” images). Please comment on dotnet-docker #5014 if you need these libraries.

Users adopting .NET 8 are the most obvious candidates for chiseled containers. You will already be making changes, so why not make one more change? In many cases, you’ll just be using a different image tag, with significant new benefits.

Ubuntu and Debian users can achieve very significant size savings over the general-purpose images you’ve had available until now. We recommend that you give chiseled images serious considerations.

Alpine users have always been very well served and that isn’t changing. We’ve also included a non-root user in .NET 8 Alpine images. We’d recommend that Alpine users first switch to non-root hosting and then consider the additional benefits of chiseled images. We expect that many Alpine users will remain happy with Alpine and others will prefer Ubuntu Chiseled now that it also has a small variant. It’s great to have options!

We’ve often been asked if we’d ever switch our convenient version tags — like 8.0 — to Alpine or (now) Chiseled. Such a change would break many apps, so we commit to publishing Debian images for those tags, effectively forever. It’s straightforward for users to opt-in to one of our other image types. We believe equally in compatibility and choice, and we’re offering both.

Summary

We’ve been strong advocates of Ubuntu chiseled containers from the moment we saw the first demo. That demo has now graduated all the way to a GA release, with Canonical and Microsoft announcing availability together. This level of collaboration will continue as we support these new images and work together on what’s next. We’re looking forward to feedback, since there are likely interesting scenarios we haven’t yet considered.

We’ve had the benefit of working closing with Canonical on this project and making these images available to .NET users first. However, we’re so enthusiastic about this project, we want all developers to have the opportunity to use chiseled images. We encourage other developer ecosystems to stongly consider offering chiseled images, like Java, Python, and Node.js.

We’ve had recent requests for information on chiseled images, after the .NET Conf presentations. Perhaps a year from now, chiseled images will have become a common choice for many developers. Over time, we’ve seen the increasing customer challenge of operationally managing containers, largely related to CVE burden. We believe that chiseled images are a great solution for helping teams reduce cost and deploy apps with greater confidence.

Category
.NETASP.NET CoreContainersLinux

Author

Richard Lander
Richard Lander
Product Manager

Richard Lander is a Program Manager on the .NET team. He works on making .NET work great in memory-limited Docker containers, on Arm hardware like the Raspberry Pi, and enabling GPIO programming and IoT scenarios. He is part of the design team that defines new .NET runtime capabilities and features. Favourite fantasy: Dune and Doctor Who. He grew up in Canada and New Zealand.

34 comments

Discussion is closed. Login to edit/delete existing comments.

Newest
Newest Popular Oldest
  • Arshad Ayub

    Hi when tried to build
    At this step: RUN update-ca-certificates getting the following error,
    runc run failed: unable to start container process: exec: “/bin/sh”: stat /bin/sh: no such file or directory
    Do I need to change something?

      • Arshad Ayub

        Hi @Richard Lander
        Thanks for response,
        We add the certs,
        Tried with the reference you attached, but Still couldn’t find the best practice to replace RUN because of multiple instructions we use (RUN).
        What can be used when we have RUN in multiple instructions in a Dockerfile?

      • Logan BussellMicrosoft employee

        Hi Arshad, I'm a maintainer on the .NET Docker team. I filed an issue that you can track for providing documentation on updating certificates in Ubuntu Chiseled: https://github.com/dotnet/dotnet-docker/issues/5045

        For your problem with multiple RUN instructions - since Ubuntu Chiseled images don't have a shell, you won't be able to chain multiple RUN instructions together with &&. You are limited to using the form which directly calls executables, as Rich mentioned above. For this reason, it's...

        Read more
      • Arshad Ayub

        Hi Logan
        Thank you for the response and giving more details

  • Kenny Pflug

    Dear .NET Team,

    thanks for the chiseled container images - they are a great improvement. I just wanted to check out the chiseled AOT images and compare them to Alpine and noticed that these are still in preview - do you have a rough estimate when they will be generally available?

    Thanks for all your efforts!

    Read more
  • johnson

    Hello Richard,

    What if want’s to use images more than 10MB ?, Also the size of container images functionality varies based on devices we use?

    Thanks in advance

    • Richard LanderMicrosoft employee Author

      I realized that I missed the opportunity in the post to describe a very important concept (which I'll make sure to cover in my next post). I'll do that now.

      and can equally be thought of as package manager tools. They just have different behavior and are intended for different use cases. The former is for a live general purpose OS and is easy to use. The latter is for building an appliance style...

      Read more
  • Dominik Jeske

    What is recommended approach for diagnosing app? – Currently during development, I sometimes log into container using shell and do some curl tests or other investigations. With those new containers this will be impossible.

    • Richard LanderMicrosoft employee Author

      One of the big benefits is that dev and prod can match. I’d suggest using regularly jammy during dev and then switch to jammy-chiseled for prod. Our OCI publish solution makes that really straightforward, with the ContainerFamily feature.

  • Qing Charles

    ARM or x64 only?

    I had to manually install the ARM .NET 8 gzips as there are no packages yet…!

  • DUBUS, Landry

    How would Ubuntu chiseled images compare to Mariner Linux distroless images?
    What would be the recommended choice between boths for deploying containers in Azure cloud?

    • Richard LanderMicrosoft employee Author · Edited

      Great question. They are similar. The Azure Linux (Mariner) images are a bit bigger.

      Our recommendation is to use the Ubuntu images. Most people are more familiar with it and it also offers a full desktop for dev (so dev and prod match). The Azure Linux images are public and work great, but are largely aimed at (and tested for) use in Azure. Ubuntu has a longer track record in terms of serving the needs of...

      Read more
  • Paulo Pinto

    Not including ICU or tzdata into the images makes them irrelevant for localized applications, unless I am missing something.

      • Paulo Pinto

        I see, thanks for the hint.

  • Señor Developer

    Hi, very cool improvements on container during the dotnet conf!. I'm testing it a bit, and the basic blazor template with the basic docker file, if I add the

    <code>

    It throws an error:

    <code>

    Any ideas what it could be? Thanks!

    Read more
      • Señor Developer · Edited

        Thanks for the quick answer, I just added docker and orchestration support for the Blazor app, then set the docker compose file as the startup project and run it pressing play.

      • Richard LanderMicrosoft employee Author · Edited

        I talked to the team. Yes, the VS tools require and that isn't changing.

        There are two options:

        I cannot in good conscience suggest adding to your chiseled image to make the tools works. That seems wrong. It's counter -productive to the intent of chiseled. You can do it, but it's kinda moraly wrong.

        A big part of this offering is that dev and prod match (if you use Ubuntu for both), so this split is...

        Read more
      • Señor Developer

        Hi Richard, thanks for the answer. I think I will go with your suggestion of targeting different containers for dev and prod. Thanks!

      • Richard LanderMicrosoft employee Author

        Understood now. I’ll ask the VS tools folks for guidance on that.

      • Señor Developer

        Really appreciate it. Thanks!

  • Marcelo Simas

    This is cool, but how would I add an OS dependency to the runtime chiseled container from mcr? It looks like you guys provide an extra tag which includes the ICU, so that is covered.

    However, we use SkiaSharp and depend on the fontconfig pkg on Alpine to be able to draw text on bitmaps so we install it using apk. How would we go about switching to chiseled runtimes?

      • Marcelo Simas

        Thanks Richard! And have a nice Thanksgiving holiday!

Read next