Recently, while building containers to be reused by various teams within a customer’s business units, we identified a security risk related to handling secrets. Within the customer’s environment, accessing external artifact resources (like those found in PyPi and Debian Packages) is blocked and instead need to go through the customer’s internal Artifactory where artifacts are cached. As a result, when building a Docker container, we need to pass access credentials to the container so that various dependencies can be installed.

It was common practice within the customer’s various applications to pass credentials needed to access the internal artifact repository as build arguments. For example:

FROM ubuntu:20.04

ARG SECRET_KEY
ENV ARTIFACTORY_SECRET=$SECRET_KEY
...

Then using --build-arg, SECRET_KEY would be passed to the container when running docker build...

The problem with using build arguments to pass secrets is the focus of this blog.

The Issue with Docker Build Time Arguments Copy link

Build-time arguments, or build args, are a convenient way to pass information into a Dockerfile during the image build process. They allow you to customize the build process based on different inputs, such as specifying versions of software components or toggling features on and off. However, this convenience comes with a hidden risk: the build arguments are stored in the image history and can be easily retrieved.

Retrieving Build Arguments via Docker History Copy link

The docker history command provides a detailed history of an image, including the commands used to build it. This includes any build-time arguments that were passed during the build process. For example, consider the following Dockerfile:

FROM ubuntu:20.04

ARG SECRET_KEY
RUN echo "Secret: $SECRET_KEY"

If you build this Dockerfile with a secret key:

docker build --build-arg SECRET_KEY=mysecretkey -t myimage .

Anyone with access to the resulting image can run docker history and see the secret key:

docker history myimage --no-trunc

The output will include a line showing the RUN command with the secret key exposed:

2 minutes ago      /bin/sh -c echo "Secret: mysecretkey"   0B

This exposure of build arguments can lead to serious security issues, especially if sensitive information such as API keys, passwords, or confidential configuration details are passed as build arguments.

The Problem with Passing Secrets as Build Time Arguments Copy link

Why It’s a Bad Idea:

• Exposure through Docker History/Inspect: As highlighted, secrets passed as build arguments are recorded in the image history, making them retrievable by anyone who can access the image.
• Insecure Storage: Even if you trust everyone who has access to the image, storing secrets in the image history violates the principle of least privilege. It’s better to minimize the number of places where sensitive information is stored.
• Lack of Control: Once an image with embedded secrets is pushed to a registry, those secrets are out of your control. Anyone who pulls the image can extract the secrets.

In our situation, the credentials used to access the artifact repository also had write permission to the artifact repository. A bad faith actor with access to the internal repository where the containers the team was building and pushing to could have retrieved the same credentials used by our build/release pipelines, impersonate the account, and pushed their own malicious code to the artifact repository. It may not be immediately obvious that someone other than the build pipeline tampered with the container. To the end consumers, the pipeline built the container.

How to Avoid Passing Secrets via Build Time Arguments Copy link

To mitigate the risks associated with passing secrets as build-time arguments, consider the following best practices:

Docker Secrets Copy link

If you need secrets at build time because you need to access protected artifacts like in our situation, Docker provides a built-in mechanism for managing secrets during the build process using Docker Build Secrets. This feature allows you to securely handle sensitive data without exposing it in the image history. Secrets are passed in at build time using the --secret flag. Secrets can be read from environment variables or files. Here’s how you can use Docker Build Secrets:

Export your secret as an environment variable:

export SECRET_KEY=mysecretkey

In your Dockerfile, you can access the secret using the --mount=type=secret flag and set to either an environment variable within the container or as a file:

FROM ubuntu:20.04

RUN --mount=type=secret,id=mysecret,env=SECRET_KEY \
    sh -c 'echo Secret: $SECRET_KEY'

NOTE: environment variables set at build-time only exist during the build process. It is also possible to mount secrets rather than using an environment variable. If you omit env=SECRET_KEY, the default file path of the secret inside the build container, is /run/secrets/<id>.

Then, you can pass secrets using the --secret flag:

docker build --secret id=mysecret,env=SECRET_KEY -t test .

NOTE: id used when passing the secret must match the id used within the Dockerfile when referencing the secret. In the example above that id reference is mysecret.

Now when you run docker history... you will no longer see the secret passed in at build time:

docker history test --no-trunc
...
13 minutes ago   RUN /bin/sh -c sh -c 'echo Secret: $SECRET_KEY' # buildkit
...

NOTE: depending on which version of Docker and BuildKit you have installed, you may need to run export DOCKER_BUILDKIT=1 to enable BuildKit and the ability to use --secret

Use Multi-stage Builds Copy link

Multi-stage builds are an effective way to hide build arguments and secrets from being exposed in the image’s build history. By breaking the Dockerfile into multiple stages, you can isolate the stages that require sensitive information and ensure that these secrets do not persist in the final image. For example, you can use an intermediate stage to fetch dependencies or perform tasks that require secrets, and then copy only the necessary artifacts to the final stage. This way, the final image does not contain sensitive information, and its build history does not reveal secrets from intermediate stages. This approach enhances security by minimizing the exposure of sensitive data. For example:

# Build stage - use secrets to install dependencies
FROM ubuntu:20.04 AS build-image

ARG SECRET_KEY
RUN echo "Secret: $SECRET_KEY"
RUN echo "hello from build stage!" > /hello.txt

# Final stage - copy dependencies installed in the previous stage
FROM ubuntu:20.04 AS runtime-image

COPY --from=build-image /hello.txt /hello.txt

Now that our Dockerfile has been converted to use the build argument SECRET_KEY in an intermediate stage rather than the final run-time stage, we will now build the image while passing our secret as a build argument:

docker build --build-arg SECRET_KEY=mysecretkey -t myimage .

Refactoring our Dockerfile to use the secret passed as a build argument to an intermediate stage to access internal repositories will result in the following output when running docker history...

docker history test --no-trunc
IMAGE               CREATED          CREATED BY                              SIZE      COMMENT
sha256:6460e02...   44 seconds ago   COPY /hello.txt /hello.txt # buildkit   6B        buildkit.dockerfile.v0
<missing>           2 months ago     /bin/sh -c #(nop)  CMD ["/bin/bash"]    0B
<missing>           2 months ago     /bin/sh -c #(nop) ADD file:748614...    72.8MB
<missing>           2 months ago     /bin/sh -c #(nop)  LABEL org.open...    0B
<missing>           2 months ago     /bin/sh -c #(nop)  LABEL org.open...    0B
<missing>           2 months ago     /bin/sh -c #(nop)  ARG LAUNCHPAD_...    0B
<missing>           2 months ago     /bin/sh -c #(nop)  ARG RELEASE          0B

As shown above, only the COPY /hello.txt /hello.txt step in the final stage is visible in the image history and none of the prior stages are visible even though we are passing the secret to the container as a build argument.

Use Environment Variables at Run-Time Copy link

Instead of embedding secrets during the build process, inject them at run-time using environment variables. This ensures that secrets are not stored in the image history. For example:

docker run -e SECRET_KEY=mysecretkey myimage

Export + Import Container Copy link

Exporting the container (not the image) is another way to flatten the image’s history. This approach requires the container to be running before exporting.

docker run test
docker export -o test.tar
docker import test.tar test:latest

Doing the above steps will result in the following output when running docker history...

docker history test --no-trunc
IMAGE              CREATED          CREATED BY   SIZE      COMMENT
sha256:792f4c...   12 seconds ago                72.8MB    Imported from -

Additional Steps Copy link

If secrets were ever exposed in the future (be it from a Docker image or accidentally leaking through other channels), there are additional steps that could be taken to prevent the impact of a rogue employee from pushing un-reviewed, malicious code:

• Sign artifacts using a private key with a tool like notation. Doing so would mean the artifact built by the pipeline would be digitally signed and verifiable by the consumer.
• Set up artifact verification on the consumer side. Before utilising the artifact, verify the artifact was signed by the pipeline which produced the artifact. If the artifact cannot be verified using a known public key, then the artifact will be prevented from being deployed. In the event of a secret leak, the rogue employee would also need the private certificate used by the pipeline to push a signed artifact.
• Do not use latest and do not rely on image tags. Tags are not immutable. They can be overwritten with a simple docker push. It is recommended to use image digest instead. If the rogue employee pushes a compromised image with the same tag produced by our pipeline, the tag will be the same, but the digest would be different. If we used digest instead of the tag, we could guarantee the image being pulled is the intended image.
• Do not give multiple permissions to a single set of credentials. The credentials used to access the internal artifact repository to install dependencies (read permission) should not be the same credentials used to push artifacts to the repository (write permission).

Conclusion Copy link

While build-time arguments in Docker offer a convenient way to customise the build process, they come with significant risks when used to pass sensitive information. By understanding the limitations and security implications of build arguments, you can avoid common pitfalls and adopt best practices to protect your secrets. Leveraging Docker Secrets, environment variables, multi-stage builds, and export/import container can help you build secure and robust Docker images without compromising sensitive data. By taking these precautions, you can enjoy the benefits of Docker while maintaining a strong security posture.

Further Reading Copy link

• Multi-stage builds
• Flatten a Docker container or image
• Build secrets
• Manage images as OCI image layout

*The featured image was generated using Microsoft Copilot’s Designer tool.