Secrets that survive the build

Most secret leaks I have written challenges for are not the obvious kind. Nobody in these scenarios pasted a password into a Dockerfile. They reached for the purpose-built mechanism, wired it up, and the secret still ended up somewhere permanent.

That is the interesting failure mode. A hardcoded credential is a mistake you can grep for. A secret that leaks through the tool designed to protect it survives review, because the review looks at the tool and the tool is correct.

Over the past couple of years I contributed a few challenges to OWASP WrongSecrets, a deliberately vulnerable app where each challenge is a real way secrets escape. Three of mine turned out to be the same shape.

1. The build secret that outlives the build

BuildKit gives you a genuinely good primitive:

RUN --mount=type=secret,id=api_key \
    ./configure --key-file=/run/secrets/api_key

The secret is mounted for exactly one instruction and never enters a layer. You can docker save the image, unpack the tarball, and it is not in there. This is the correct answer, and it is the answer people are given when they ask.

Then someone needs the value later in the build, so the RUN step does this instead:

RUN --mount=type=secret,id=api_key \
    cp /run/secrets/api_key /app/secret.txt && ./configure

The mount was still temporary. The copy was not. /app/secret.txt is now a regular file in a regular layer, shipped to every registry the image reaches and readable by anyone who can docker run it.

That is Challenge 52. What makes it a good challenge rather than a cheap one is the second half: the build script that did the copying lives in the repository, in CI, in public. So the image tells you the secret exists and the repo tells you how it got there.

The lesson is narrower than "don't leak secrets". It is that --secret protects the transport, not the value. Once your build step has the plaintext in hand, every guarantee is back on you. Deleting the file in a later layer does not help either, since the earlier layer is still in the image; the delete just adds a whiteout on top of it.

2. The secret file that was in the repo the whole time

Docker Compose has a first-class secrets block, and it is easy to feel finished once you have used it:

services:
  db:
    image: postgres
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secretfiles/db_password.txt

Nothing here is wrong. The secret is mounted at /run/secrets/db_password rather than passed as an environment variable, which is the improvement you wanted, because env vars show up in docker inspect, in crash dumps, in anything that logs the process environment.

But file: is a path, and paths point at files, and that file has to exist for docker compose up to work. So it gets committed. Now the credential is in git history, in every clone, in every fork, on the laptop of everyone who ever checked the project out, and it is still there after you delete it in a later commit.

That is Challenge 51. The compose file is a model citizen. The repository is the problem, and the compose file is what made keeping the file in the repository feel reasonable.

Adding the path to .gitignore fixes the next commit and none of the previous ones. If a value like this was ever pushed, the only real remedy is rotation. Rewriting history helps, but you have to assume it was already read.

3. The sealed secret with the key next to it

Bitnami Sealed Secrets solves a genuinely awkward problem. A Kubernetes Secret is base64, not encryption, so it cannot go in git. A SealedSecret is asymmetrically encrypted to a controller running in the cluster, so it can, and the whole point is that a public repository is a safe place for it.

Which is true, right up until the private key is in the same repository. That is Challenge 48: a sealed-challenge48.json sitting next to the main.key that decrypts it. The ciphertext is safe to publish. The key is not, and their being adjacent is what makes it painless to write the commit that includes both.

Sealed Secrets are worth using, but they are worth being honest about too. They are still hardcoded secrets, just encrypted ones, which means you get no audit trail of who read what and from where. And rotating the sealing key means re-sealing every secret encrypted with it, potentially across repositories you will have to go find.

The shape they share

Each of these has the same structure:

• The mechanism moved the secret from an unsafe place to a safe place.
• Something else, a copy, a source file, a key, stayed behind in the unsafe place.
• Review looked at the mechanism, and the mechanism was correct.

So the question that finds these is not "are we using secrets properly". It is what does this mechanism need in order to work, and where does that thing live? BuildKit needs plaintext inside the build step. Compose needs a file on disk. Sealed Secrets needs a private key somewhere. Each requirement is a place to look.

What I actually check

A short list, in rough order of how often it finds something:

1. docker history --no-trunc <image> for build args and inline values.
2. Unpack the image and grep the layers, not just the running container. docker save and searching each layer tarball catches files that were created and then deleted.
3. Every file: and path: in compose and Helm values, then git log on whatever they point at.
4. Anything in the repo with the extension .key, .pem, or .p12, and a look at what it decrypts.
5. kubectl get secret -o yaml on a namespace, remembering that base64 is not encryption and anyone with read access on that namespace has the value.

None of this is sophisticated. That is sort of the point: these secrets are not hidden behind clever exploitation, they are sitting in a layer or a file waiting for someone to look, and the reason nobody looked is that the tooling above them was doing its job correctly.

If you want to try the challenges rather than read about them, WrongSecrets runs locally in a container and the ones above are 48, 51 and 52.