Proposal for support for runtime composefs validation by alexlarsson · Pull Request #28658 · containers/podman

alexlarsson · 2026-05-06T13:40:10Z

In the automotive sphere, we're interested in having some level of runtime validation. For the rootfs we already get this from bootc using composefs. However, if we're also using containers in /var/lib/containers, those are not protected by this. However, containers/storage already (optionally) supports composefs, so we should be able to do something similar for podman.

Here is what this MR, and the related changes in container-libs does:

Add option --security-opt signature=[check,require]. If this is set, then the signature for the manifest is validated at podman run time, which allows us to trust the manifest data, like the annotations.
Add support in the overlay driver to pass in a set of expected composefs fs-verity digests for the layers. When these are used, it will validate at mount time that all the layers have an expected digest, and ensures the overlayfs mount is mounted with verity=require (which requires all files in the mount to be backed by a file with a fs-verity digest from the composefs blob).
If --security-opt verity=enforce is passed to podman run, then podman looks at the per-layer annotations in the image manifest (which we ideally trust due to a signature) for io.containers.composefs.digest keys, where you can give a list of allowable fs-verity digests. These are then forwarded to the overlayfs driver which validates this at mount time.

With the above, we can have a pretty robust validation of the container at runtime. There are some weak points:

An attacker could replace the podman binary or some dependency to not do the checks
An attacker could change the signature policy
You have to start podman with the right arguments

All of these are fixable in a controlled environment. For example, if you have a read-only /usr and /etc like bootc, and you ship a quadlet file in /usr that has the right arguments, then you can have some trust in that the right code is running, and you can do "podman pull" to get a new image version, keeping this trust.

I have an example signed image with annotations at https://quay.io/repository/alexl42/centos-verity. See the description there for the public key used. With I can run a validated image:

# podman run --security-opt signature=require --security-opt verity=enforce -ti quay.io/alexl42/centos-verity:10 echo foo

You can also see how it works with an unsigned and no-verity image:

# podman run --security-opt signature=require --security-opt verity=enforce -ti quay.io/centos/centos:9 echo foo
Error: manifest signature verification failed: No signature verification policy found for image docker:quay.io/centos/centos:9
# podman run --security-opt verity=enforce -ti quay.io/centos/centos:9 echo foo
Error: verity enforcement: layer 0 missing io.containers.composefs.digest annotation

Or if i tweak the composefs blob:

# podman run --security-opt signature=require --security-opt verity=enforce -ti quay.io/alexl42/centos-verity:10 echo foo
Error: mounting storage for container 45b43778da976b9997ace9eda98b17ac4c87c4e2f820c59a07f9c4f7429e884d: composefs blob /mnt/data/containers/storage/overlay/433ca53ed3a7208fc5834c1ee92839b01ed5ff7fccef4e80c0437992c4806267/composefs-data/composefs.blob has fs-verity digest "sha256:f86904b89e9d4b5c5436a657647c3d0309be742c5f9460b786027fb2968ea167", not in allowed list

I don't expect this PR to just be necessarily merged as is, but I'd like to bring this up for discussion. We'd like a feature like this in automotive, is that reasonable? Is the approach reasonable? Is the interface reasonable?

Discussion points:

Do we maybe also want config options for making these checks default?
Is using layer annotations right? They are sort of a pain to set.

Some minor notes:

We need to support multiple for forward compatibility if we support e.g. other fs-verity hashes or composefs versions.
This depends on composefs: Properly add verity info to generated dump file container-libs#816
In the work on OCI sealing in the next gen composefs-rs there is some similar annotations for this, although those are in a separate artifact. We should ensure that these don't conflict, and are properly versioned for future use of this in podman.

pypingou · 2026-05-07T11:53:50Z

Do we maybe also want config options for making these checks default?

I think we would like this yes, especially as we know that not everyone will use quadlet files in /usr

mheon · 2026-05-08T17:54:32Z


 		options = append(options, libpod.WithRootFSFromImage(newImage.ID(), resolvedImageName, s.RawImageName))
+
+		if s.SignaturePolicy != "" {


I feel like there's a TOCTOU risk doing this in Specgen. The container isn't created yet, if the image tag we want to use is replaced before it is this check is subverted

To add to this anything in specgen is by design not doing runtime validation, specgen runs once when the container is created. For podman stop/start it will not be called again.

Yes that does not matter for the quadlet use case but still if such cli options exists it must work with all of podman not just quadlet.

ok nvm I fully read the code now, I think the TOCTOU does not matter because you pass in the verify digests and if the image was replaced in between then at mount time the digests will be invald and cause failure as they should.

Yes, that main requirement is that we validate the manifest json at some point, and then make any security relative decisions based on the manifest content (such as the fs-verity deltas) only from exactly the same data that we validated.

mheon · 2026-05-08T17:56:59Z

+
+const verityDigestAnnotation = "io.containers.composefs.digest"
+
+func extractVerityDigests(imageData *libimage.ImageData) ([][]string, error) {


A further reason not to do this in Specgen: You don't know what storage driver is in use by Libpod, so you could try to run this check on a system using a btrfs or zfs store. From my initial read of the code it'll probably work but you definitely aren't getting the benefits you expect if composefs isn't the backing store.

Luap99 · 2026-05-11T15:43:45Z

Can you expand more on your security/thread model please?

Right now the digest is looked up once then put into the container config (db) before it is passed to the storage mount, you say you use a read only fs but the db of course must be writeable so the attacker could try to write to the sqlite db to unset these container config fields.

For policy verification we also support loading that via env var that points to any file so that could also be used as attacker if they can set this for the podman process.

Regarding quadlets are only in ro /usr/etc I am not sure that alone matters. We also lookup /run/containers/systemd/ which I guess would be writeable or even systemd itself uses run/systemd/system which could be used to overwrite the service with another unit.

So I am not sure the controlled env will help you that much. If the attacker can get root with write access on the system there is nothing we could do.

alexlarsson · 2026-05-18T08:45:22Z

Its been a while since I looked at the podman codebase, so I marked this Draft for precisely the reason that I'd like some highlevel review on where this should go, and you're right that there is a potential issue with the config being stored in a db post-validation. Doing the validation later seems better from this PoV, but then it may be harder to do the actual validation.

Anyway, you correctly ask for a threat model, and I can at least give you what the automotive version of this is:

Suppose you have a sealed system, that verifies (secure-boot style) at boot and runtime:

The kernel
The kernel command line
The initrd
The composefs rootfs (against per-build key in initrd)
Uses a transient /etc (i.e. a per-boot tmpfs backed on /usr/etc from the composefs)
Ships podman in the read-only root, configured to use a transient store.
/var is writable in a persistent way

In such a setup we can trust everything except /var, at least from a clean boot. This is good, because the goal are:

Protect against accidental changes to the system (hw issue, damage, fs bugs, whatever)
Protect against a deliberate attacker gaining root access to persist this across boot. (Ideally we also protect against root attacks in the first place, but this is a separate defense-in-depth concern).
Disallow end users to run modified software in their cars.

We already support embedding containers in the system image (as separate image store directory), which gives use some of the above features for containers. However, we would like to also extend this to being able to install and update containers in /var, separate from the bootc image. This will allow faster and partial OTA updates with less risk. And, we would like to keep the above features and goals for apps using such containers.

So, lets assume we can "trust" /usr, /etc, $PATH, etc. What can we do to ensure that we can also install container images in /var and when we run them, we will run the code that was intended. And, this model should include hostile root-running code trying to persist root-rights across reboot.

As the most basic example, lets assume we have a quadlet in the trusted read-only bootc image. This means we have a trusted podman binary, a trusted podman config, and a trusted podman run commandline, but an untrusted /var/lib/containers. The storage config enables the overlayfs backend and the features to allow composefs images and makes the container root read-only by default (and this is not overridden in the run command). Can we, in this world, give the above guarantees? The hope is that we can run only trusted code and config up to the container being started, and have the container image content be validated using composefs digests that are trusted due to being referenced in a signed json manifest that map to a key that is part of the read-only trusted rootfs.

The main weakness I see is that an individual container may use a volume to persist data, and an attacker could modify it to attack the container, and then further use some container escape to get root access on the host. But, ignoring that vector, I think this is doable, although my initial draft may be naive in some aspects.

Luap99 · 2026-05-18T11:03:33Z

As the most basic example, lets assume we have a quadlet in the trusted read-only bootc image. This means we have a trusted podman binary, a trusted podman config, and a trusted podman run commandline, but an untrusted /var/lib/containers. The storage config enables the overlayfs backend and the features to allow composefs images and makes the container root read-only by default (and this is not overridden in the run command). Can we, in this world, give the above guarantees? The hope is that we can run only trusted code and config up to the container being started, and have the container image content be validated using composefs digests that are trusted due to being referenced in a signed json manifest that map to a key that is part of the read-only trusted rootfs.

Thanks, given that I think the current approach sounds reasonable if we move more of the validation into libpod (container start time) and then do not store the annotations as part of the container config, once validated with policy.json they need to passed along the call stack in memory to the storage mount code IMO. I have no real opinion on the image design questions, i.e. layer annotations for the hashes.

Of course once an attacker gains root on the running system they could turn of the policy.json verification and/or overmount /usr/bin/podman or the quadlet file.

alexlarsson · 2026-05-19T06:53:51Z

I think the current approach sounds reasonable if we move more of the validation into libpod (container start time) and then do not store the annotations as part of the container config, once validated with policy.json they need to passed along the call stack in memory to the storage mount code

@Luap99 That makes sense. However, I don't think there is a way to communicate commandline options to libpod other than via the db currently, right? Basically, all the "podman run" commands end up in the "podman create", and we don't have anything from "podman start" that can affect things. So, this means we have to either add a way to forward such options, or instead make the "need to be signed" a global config option instead. Opinions?

Luap99 · 2026-05-19T12:53:32Z

I think the current approach sounds reasonable if we move more of the validation into libpod (container start time) and then do not store the annotations as part of the container config, once validated with policy.json they need to passed along the call stack in memory to the storage mount code

@Luap99 That makes sense. However, I don't think there is a way to communicate commandline options to libpod other than via the db currently, right? Basically, all the "podman run" commands end up in the "podman create", and we don't have anything from "podman start" that can affect things. So, this means we have to either add a way to forward such options, or instead make the "need to be signed" a global config option instead. Opinions?

Sorry I think I was not clear, yes you are right we need a db option to say enforce signatures. But that should just be like the cli option basically and not store the digests IMO.

Then on container start in libpod we check the container config if signature must be checked then get manifested validated with policy.json signatures, get the layer digests and then pass them to the mount calls, basically keep the signature validation right before the mount time.

I agree that technically an attacker could still just unset the option from db config if they get write access to the db so it is not much different in that sense but at least the digest values are guaranteed to be validated each time before use.
I think using a global containers.conf option might make sense here for sure, if you know you want this for all containers on the system without being able to turn this off per container then just having this might even be cleaner because whoever configures the quadlets cannot just forget to set it one of them.

pypingou · 2026-05-19T13:04:03Z

I think using a global containers.conf option might make sense here for sure, if you know you want this for all containers on the system

This got me wondering: how will this work for the container images that are included in the ostree commit? Are we going to run them w/ composefs on top of a container storage that's already mounted through ostree w/ composefs? (not saying this would be a problem, just curious if that'd be the idea)
If that is correct, I wonder if this would have an impact on performances

alexlarsson · 2026-05-19T16:41:42Z

I think using a global containers.conf option might make sense here for sure, if you know you want this for all containers on the system

This got me wondering: how will this work for the container images that are included in the ostree commit? Are we going to run them w/ composefs on top of a container storage that's already mounted through ostree w/ composefs? (not saying this would be a problem, just curious if that'd be the idea) If that is correct, I wonder if this would have an impact on performances

We could maybe tie this to the container image source (i.e. what directory it came from)? Initally I was thinking we could link it to the name of the image, and then to the signature policy for that name. However, the name is not signed, only the digest content, so maybe that won't be secure.

This is work needed for supporting composefs + fs-verity

This checks, as late as possible so its all in-memory, a check that the manifest for the image being run is signed per the policy, and if the signature is valid passes on the expected fs-verity digest for the image layers. Currently these are enabled with `--security-opt signature=check/require`, and `--security-opt verity=enforce`. But, note that these options are just persisted in the container storage config for the container. To be completely secure, these should in the future be made a global config option to able to ensuse this cannot be overridden.

This is a small script that rewrites a manifest to add the fs-verity annotations, converts to zstd::chunked, signs it with cosign and pushes it to a repo.

alexlarsson · 2026-05-26T08:44:07Z

Ok, I changed the composefs-verity branch of alexlarsson/container-libs a bit. Now we pass the fs-verity expected digests at mount time in a new MountWithOptions() store call, rather than persiting the expected config on disk. This should make things safer against an attacker being able to write to /var/lib/containers.

Note: The podman code still uses options like --security-opt signature=require --security-opt verity=enforce, to enable this, which is not strictly safe, as these will be persisted in the containers storage in "create" and read back in "start". However, it lets you test this stuff easily, and if the overall approach is ok we can easily add global options to enforce this for every container later.

I tested this by configuring:
storage.conf:

[storage.options.pull_options]
enable_partial_images = "true"
use_hard_links = "true"
convert_images = "true"

[storage.options.overlay]
use_composefs = "true"

/etc/containers/registries.d/quay.io.yaml

docker:
  quay.io:
    use-sigstore-attachments: true

policy.json:

	    "quay.io/alexl42/centos-verity": [            
		{
                    "type": "sigstoreSigned",
                    "keyPath": "/etc/containers/keys/verity-test.pub"
		}
	    ]

verity-test.pub:

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE9N/z8zbWF5CnS1cEfzov+3cH3lRC
eeJyaiamVDVgGJ3De0FMse4YqavYvpVMHGKyGaRBrDh2OrNzVE7DCpGbNQ==
-----END PUBLIC KEY-----

Then I can:

$ sudo podman pull quay.io/alexl42/centos-verity:10
$ sudo podman run --security-opt signature=require --security-opt verity=enforce -ti quay.io/alexl42/centos-verity:10 echo foo
foo

If you run with debug you will see:

DEBU[0000] Validating manifest signature of cde67af27a1fcaeeeb849e5993e63d9f418a5a30caa4ab4d95709fd102e0a879, requireSigned=true 
DEBU[0000] IsRunningImageAllowed for image docker:quay.io/alexl42/centos-verity:10 
DEBU[0000]  Using transport "docker" specific policy section "quay.io/alexl42/centos-verity" 
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0000] Inspecting image cde67af27a1fcaeeeb849e5993e63d9f418a5a30caa4ab4d95709fd102e0a879 
DEBU[0000] exporting opaque data as blob "sha256:cde67af27a1fcaeeeb849e5993e63d9f418a5a30caa4ab4d95709fd102e0a879" 
DEBU[0000] exporting opaque data as blob "sha256:cde67af27a1fcaeeeb849e5993e63d9f418a5a30caa4ab4d95709fd102e0a879" 
DEBU[0000] Expected verity digests: [[sha256:55731366fad55cb15df3d3892480dd706ee78b3e2acadfb0f48c69a2c03bc4b6]]

And if you tweak the digest for the composefs blob you get things like:

$ sudo odman --security-opt signature=require --security-opt verity=enforce -ti quay.io/alexl42/centos-verity:10 echo foo
Error: mounting storage for container b1dfce79527c7e98336b3343ae5e97a8a014c08f8268f689b028b80dbcc6bcea: composefs blob /var/lib/containers/storage/overlay/433ca53ed3a7208fc5834c1ee92839b01ed5ff7fccef4e80c0437992c4806267/composefs-data/composefs.blob has fs-verity digest "sha256:2f49c9f9f6445ad05e34da360b9dfed49274b1a9080be25016a69f73a04806cd", not in allowed list

alexlarsson · 2026-05-26T08:44:49Z

If this seems like a workable approach, I'll start with making a PR for container-libs with the support code.

alexlarsson marked this pull request as draft May 6, 2026 13:40

mheon reviewed May 8, 2026

View reviewed changes

alexlarsson added 3 commits May 26, 2026 10:00

WIP: Vendor in the composefs-verity branch of alexlarsson/container-libs

f5514d4

This is work needed for supporting composefs + fs-verity

Add contrib/add-verity-annotations helper script

da0d07a

This is a small script that rewrites a manifest to add the fs-verity annotations, converts to zstd::chunked, signs it with cosign and pushes it to a repo.

alexlarsson force-pushed the composefs-verity branch from 01f4e91 to da0d07a Compare May 26, 2026 08:30


		options = append(options, libpod.WithRootFSFromImage(newImage.ID(), resolvedImageName, s.RawImageName))

		if s.SignaturePolicy != "" {


		const verityDigestAnnotation = "io.containers.composefs.digest"

		func extractVerityDigests(imageData *libimage.ImageData) ([][]string, error) {

Conversation

alexlarsson commented May 6, 2026

Uh oh!

pypingou commented May 7, 2026

Uh oh!

mheon May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Luap99 May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Luap99 May 11, 2026

Choose a reason for hiding this comment

Uh oh!

alexlarsson May 18, 2026

Choose a reason for hiding this comment

Uh oh!

mheon May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Luap99 commented May 11, 2026

Uh oh!

alexlarsson commented May 18, 2026

Uh oh!

Luap99 commented May 18, 2026

Uh oh!

alexlarsson commented May 19, 2026

Uh oh!

Luap99 commented May 19, 2026

Uh oh!

pypingou commented May 19, 2026

Uh oh!

alexlarsson commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexlarsson commented May 26, 2026

Uh oh!

alexlarsson commented May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alexlarsson commented May 19, 2026 •

edited

Loading