WIP: Rough draft for updated generic OCI sealing by cgwalters · Pull Request #226 · composefs/composefs-rs

cgwalters · 2026-02-08T22:44:50Z

This is just some rough draft raw material that builds on:

oci: Add OCI-native manifest/config storage and image management #216 (🆕 now requires manifest/config as external not inline splitstream objeects, also the OCI artifact and referrer support was key)
doc/plans: Update OCI sealing spec (kernel sigs, flattened layers) #224 (Lots of updates there, I redid to go all-in on an external signature+digests by default)

cgwalters · 2026-02-12T16:53:07Z

+                    composefs_oci::signing::FsVeritySigningKey::from_pem(&cert_pem, &key_pem)?;
+
+                // Build subject descriptor from the source image's manifest
+                let manifest_json = img.manifest().to_string()?;


Hmm we actually need to operate on the raw original representation, can't rely on to_string() always giving us the same thing.

cgwalters · 2026-02-12T16:53:56Z

+        /// Image reference (tag name)
+        image: String,
+        /// Path to the OCI layout directory (must already exist)
+        oci_layout_path: PathBuf,


I think we can use clap(value_parser) into an ocidir directly or so

cgwalters · 2026-02-12T16:54:28Z

-/// the container to be mounted with integrity protection.
-///
-/// Returns a tuple of (sha256 content hash, fs-verity hash value) for the updated configuration.
-pub fn seal<ObjectID: FsVerityHashValue>(


Might be cleaner if we do a prep commit that removes the old sealing as we know we're not going to do it anymore.

cgwalters · 2026-02-12T16:54:45Z

+/// # Returns
+///
+/// The number of referrer artifacts exported.
+pub fn export_referrers_to_oci_layout<ObjectID: FsVerityHashValue>(


Something like this could land as a prep commit

cgwalters · 2026-02-12T16:54:50Z

+    use std::fs;
+    use std::io::Write;
+
+    let blobs_dir = oci_layout_path.join("blobs").join("sha256");


cgwalters · 2026-02-12T16:56:27Z

+        format!("{seed:02x}").repeat(32)
+    }
+
+    fn sample_subject() -> Descriptor {


Let's unify this stuff with shared infra to generate an ocidir with known content

cgwalters · 2026-03-11T12:38:12Z

This one will need to logically depend on #225 because that one has a lot of hardening for the EROFS parser

Add set_write_concurrency() to Repository for overriding the default parallelism. Add read_filesystem_with_semaphore() as a public entry point that accepts an explicit Semaphore, and refactor the internal read_filesystem_impl() to centralize semaphore selection. Prep for wiring up --threads in mkcomposefs. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

The patch recipe referenced crates/cfsctl which was never a valid path; the crate has always been named composefs-ctl. Also relax the clean-tree check to allow untracked files (only committed changes need to match the pinned revision). Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

import_oci_layout() was opening the layout directory before calling ensure_writable(), so pulling into a read-only repo produced a misleading ENOENT error instead of a clear 'not writable' message. Move the write check to the top of the function, matching the existing skopeo pull path. Fixes privileged_pull_readonly_repo integration test. Signed-off-by: Colin Walters <walters@verbum.org>

For compatibility with the C composefs, we need to support writing directly to a flat XX/DIGEST path, without a leading `objects/`. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

The script hardcoded /usr/share/edk2/ovmf/OVMF_CODE.fd which is only present on Fedora. Probe a list of common paths (Ubuntu's ovmf package uses /usr/share/ovmf/OVMF.fd, Arch uses /usr/share/edk2/x64/OVMF.4m.fd) so the script works across distros without manual adjustment. Also add -machine q35, required on newer QEMU builds (e.g. RHEL10/CentOS Stream 10) where the default pc-i440fx machine type doesn't pair well with OVMF for EFI boot. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

The combined OVMF.qemuvars.fd with -bios hangs indefinitely on RHEL10/ CentOS Stream 10 QEMU (qemu-kvm 9.x). Use the split OVMF_CODE.fd + OVMF_VARS.fd files with -drive if=pflash and -machine q35 instead, which works correctly. Fall back to -bios with the combined image on distros that only ship the combined file (Ubuntu, Arch). Updated both testthing.py (which drives the example integration tests) and the fix-verity helper script (which runs the in-VM verity fixup pass). A temporary copy of OVMF_VARS.fd is made so UEFI can write to it without modifying the original system file. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

composefs-setup-root validates that the repo's meta.json has fs-verity enabled before trusting the repo. The dracut hook was only enabling verity on the content objects, so setup-root would see the repo as insecure and refuse to proceed. Switch the working directory to /sysroot/composefs (instead of the objects subdirectory) so we can enable verity on meta.json in addition to all the content objects. Also quote the loop variable and use the full relative path for clarity. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

The 30s default is tight on slower hardware (e.g. CentOS Stream 10 with OVMF pflash init overhead) — the VM boots successfully but just barely misses the window. 60s gives enough headroom while still being short enough to catch genuinely broken VMs. CI on Ubuntu with KVM acceleration boots well under 30s so the extra budget costs nothing. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

…info CLI Add support for generating V1 EROFS images compatible with the C composefs tools (mkcomposefs/composefs-info 1.0.8+). V1 uses compact inodes, BFS layout, and a simpler on-disk structure. Adds --erofs-version flag to cfsctl, new mkcomposefs and composefs-info compatibility subcommands, and RepositoryConfig for cleaner repo initialization. Note: this commit does not compile with --features oci (the default) until the following commit migrates OCI crate callers. Assisted-by: OpenCode (Claude Sonnet 4.5) Signed-off-by: Colin Walters <walters@verbum.org>

…ne support Migrate OCI crate callers to the new RepositoryConfig API and add dual-format (V1+V2) EROFS image generation during OCI pull. Add the composefs.digest= karg for V1 EROFS images and update boot integration to generate the appropriate karg based on repository format version. Assisted-by: OpenCode (Claude Sonnet 4.5) Signed-off-by: Colin Walters <walters@verbum.org>

Library crates composefs-oci and composefs-http previously created indicatif progress bars directly, coupling them to terminal output. But callers like bootc really need structured metadata so they can render their own progress. Closes: composefs#140 Generated-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

Adds support for generating V1-format EROFS images alongside the existing V2 ("composefs") format. V1 EROFS is the on-disk format used by the C composefs implementation, enabling interoperability between the Rust and C stacks. Eventually, the idea is we deprecate the C implementation and replace it with this. It turns out that the EROFS filesystems we were generating can't be mounted by RHEL9 era kernels. So that's another reason to fix this. However: we can't change the EROFS layout we output by default, because current sealed UKIs basically require compatibility. So: Let's thread through the concept of versioning here. While we're doing this, the idea is that for sealed UKIs, we will distinguish "container image wants v1 format" by detecting a new `composefs.cmdline=` karg. Otherwise, the repository now defaults to v1 for new repos. Otherwise, it defaults to generating both versions. Existing repositories can turn off the V2 format as well. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

Implement the containers-storage import path (cstor module) which can import OCI images directly from podman/buildah storage without going through skopeo, using reflinks or hardlinks to avoid data copies when the composefs repo is on the same filesystem. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>

Pulling into a read-only repository previously failed deep inside the tar splitting pipeline with confusing errors. Add a faccessat(W_OK) pre-flight check at the top of pull_image() so the user gets a clear 'Repository is not writable' message before any network or I/O work begins. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>

We didn't have good converage of this before at the unit testing level. This builds on top of our prior dumpfile based test fixture. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>

Bootc needs both a plain EROFS image (for composefs mounts) and a boot-transformed EROFS (with /boot emptied, SELinux labels applied). This commit adds the bootable variant as a second named ref on the config splitstream, using BOOT_IMAGE_REF_KEY ("composefs.image.boot") alongside the existing IMAGE_REF_KEY ("composefs.image"). The same cascade rewrite pattern applies: adding a boot EROFS ref rewrites config -> manifest -> tag, and GC keeps the boot EROFS alive through the config ref chain. CLI: - cfsctl oci pull --bootable - cfsctl oci mount --bootable Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

The biggest goal here is support for Linux kernel-native fsverity signatures to be attached to layers, which enables integration with IPE. Add support for a fully separate OCI "composefs signature" artifact which can be attached to an image. Drop the -impl.md doc...it's not useful to try to write this stuff in markdown. The spec has some implementation considerations, but it's easier to look at implementation side from a code draft. Add standardized-erofs-meta.md as a placeholder document outlining the goal of standardizing composefs EROFS serialization across implementations (canonical model: tar -> dumpfile -> EROFS). Assisted-by: OpenCode (Claude Opus 4.5) Signed-off-by: Colin Walters <walters@verbum.org>

Implement end-to-end support for cryptographically signing composefs OCI images using PKCS#7/fsverity detached signatures, stored as OCI referrer artifacts following the 'composefs erofs-alongside' spec. Core signing infrastructure (composefs crate): - Add fsverity algorithm constants and ComposeFsAlgorithm type - Add formatted_digest module for kernel-compatible fsverity digest construction (the 12-byte header + raw hash used by the kernel's FS_IOC_ENABLE_VERITY ioctl) - Add kernel keyring support via composefs-ioctls keyring module (inject X.509 certs into .fs-verity keyring for kernel-level signature enforcement) OCI signing library (composefs-oci crate): - signing.rs: FsVeritySigningKey (sign) and FsVeritySignatureVerifier (verify) using openssl PKCS#7 with DETACHED|BINARY|NOATTR flags, compatible with Linux kernel fsverity builtin signature verification - signature.rs: OCI artifact manifest builder/parser for the 'application/vnd.composefs.erofs-alongside.v1' artifact type, storing per-layer and merged EROFS images alongside their PKCS#7 signatures as typed layers with composefs.* annotations - image.rs: compute_per_layer_digests() and compute_merged_digest() for deterministic EROFS image generation from OCI layer stacks - oci_image.rs: seal_image() to compute and embed the composefs fsverity digest into the OCI config, export/import to OCI layout directories (migrated to ocidir crate for atomic I/O), referrer index management CLI commands (cfsctl): - 'oci seal <image>' — compute composefs EROFS, embed fsverity digest - 'oci sign <image> --cert --key' — create signature artifact - 'oci verify <image> [--cert]' — verify signatures (digest-only without --cert, full PKCS#7 with --cert) - 'oci mount <name> <mountpoint> [--require-signature --trust-cert]' — verify signatures before kernel mount - 'oci pull ... --require-signature --trust-cert' — verify after pull - 'oci push <image> <dest> [--signatures]' — export to OCI layout - 'oci export-signatures <image> <dest>' — export just artifacts - 'oci inspect' — show referrer info in JSON output - 'keyring add-cert <pem>' — inject cert into kernel keyring The mount and pull --require-signature paths share a common verify_image_signatures() helper that recomputes expected EROFS digests and verifies each PKCS#7 signature blob against the trusted certificate. The mount command now also resolves tag names (via OciImage::open_ref) instead of requiring raw config digests, consistent with seal/sign/ verify. Integration tests: - signing.rs: 17 unprivileged tests covering sign, verify, wrong cert, export, seal+sign roundtrip, artifact structure, --require-signature on pull and mount - privileged.rs: 7 tests for real fsverity enforcement, kernel keyring injection, kernel signature acceptance/rejection - podman.rs: 3 tests building real container images via podman - cli.rs: updated for richer OCI test layout (4 entries) and new oci push/roundtrip tests - test-oci-sign-verify.sh: standalone shell-based integration tests Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

Build cfsctl in release mode with pre-6.15 and oci-client features, then push the binary to GHCR as an OCI artifact tagged by branch name and short SHA. This lets the sealed demo Containerfile fetch a pre-built binary via oras instead of building from source. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>

CentOS Stream 10's kernel 6.12 doesn't support direct file-backed EROFS mounts (added post-6.12 in mainline). The rhel9 feature enables loopback device creation for EROFS images, which is needed for the composefs overlay mount to work. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>

Upgrade ocidir from 0.6 to 0.7 (which includes OCI artifact manifest support from bootc-dev/ocidir-rs#64, though that hasn't shipped in a release yet). This eliminates the duplicate ocidir dependency. Downgrade oci-client from 0.16 to 0.15 since 0.16 pulls in oci-spec 0.9 while everything else (containers-image-proxy, ocidir, composefs-oci) uses oci-spec 0.8. With this change, only a single oci-spec version (0.8.4) appears in the dependency tree, eliminating the need for the JSON round-tripping bridge in referrers.rs. Also upgrade cap-std-ext 4 -> 5 to match ocidir 0.7's dependency, and remove unused cap-std-ext from cstorage (it was declared but never imported). Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

Using std::process::exit bypasses normal error handling and skips destructors. Every other error path in the CLI uses anyhow::bail. Also switch the verify_raw error handling from .is_err() to a match statement to make the control flow more explicit. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

The package was renamed from fsverity-utils to fsverity in current Debian unstable. Update the test dependency script accordingly. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

Resolve compilation errors introduced when rebasing sealing-impl onto composefs-c-compat, which had several API changes: - oci-spec upgraded 0.8->0.9, add features=["image"] to Cargo.toml deps - compute_image_id() now takes FormatVersion::V1 argument - write_config() now takes image_ref_v1 and boot_image_ref_v1 args - write_manifest() / rewrite_manifest() take Vec<(K,V)> not HashMap - Repository::init_path() takes RepositoryConfig instead of (algo, bool) - mkfs_erofs moved to crate::erofs::writer::mkfs_erofs Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

Signed-off-by: Colin Walters <walters@verbum.org>

Several integration tests added in the sealing-impl PR broke after rebasing onto composefs-c-compat, because: 1. Missing init_insecure_repo: Tests in signing.rs, parts of cli.rs, and podman.rs created bare tempdirs and immediately ran oci pull or create-image without first initializing the repo. The old code apparently auto-inferred metadata from an objects/ dir; the new code requires explicit 'cfsctl init'. Fix by moving init_insecure_repo to main.rs as pub(crate) and calling it at the start of every affected test. 2. Sign/verify digest mismatch: compute_per_layer_digests and compute_merged_digest hardcoded FormatVersion::V1, but the sign command uses generate_per_layer_images/generate_merged_image which use mkfs_erofs() (V2 default). Fix by using repo.erofs_version() throughout so sign and verify always agree on which EROFS format the current repository uses. 3. OCI_LAYOUT_COMPOSEFS_ID / V1_ID constants: ocidir 0.7 serialises OCI configs differently from 0.6, so the pinned fs-verity digests of the deterministic test image changed. Updated both constants to the values produced by the current code. 4. test_layer_tar_roundtrip expected 2 entries but create_oci_layout creates 4 (usr/, usr/bin/, etc/, usr/bin/hello.txt). Updated the assertions to match the actual layout. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

The privileged_sign_and_verify_with_verity, privileged_seal_then_sign, and privileged_keyring_and_verify_with_verity tests were pulling into a verity repo without first initializing it, causing a 'no meta.json' error. Add `cfsctl --repo {repo} init` (without --insecure) before the pull call in each test. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

All four functions involved in sign/verify (generate_per_layer_images, generate_merged_image, compute_per_layer_digests, compute_merged_digest) now unconditionally use FormatVersion::V1 instead of repo.erofs_version(). V2 is a local efficiency optimization; signed artifacts pushed to a registry must use V1 so any verifier can check them regardless of its own repo configuration. Using repo.erofs_version() would cause sign/verify to produce mismatched digests on repos initialized with V2 format, breaking verification. Also add a tag parameter to generate_boot_image so the caller can update the tag to point at the new (boot-containing) manifest after it is rewritten. Without this, inspect via the original tag would show composefs_erofs but not composefs_boot_erofs after a --bootable pull. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

Two related bugs fixed: 1. After 'cfsctl oci pull --bootable', inspect via the image tag showed composefs_boot_erofs but not composefs_erofs. The cause: pull returns the original pre-rewrite manifest digest, but ensure_oci_composefs_erofs (called inside pull) produces a new manifest digest. generate_boot_image was called with the old digest whose config lacks image_ref_v1, so ensure_oci_composefs_erofs_boot preserved None into the boot manifest's config. Fix: resolve the tag after pull to get the current (rewritten) manifest digest before calling generate_boot_image. 2. privileged_kernel_accepts_valid_signature assumed that adding a cert to the kernel .fs-verity keyring at runtime would make it trusted for FS_IOC_ENABLE_VERITY --signature. Some kernels require certs to chain to a built-in trust anchor and reject self-signed runtime certs even after a successful keyctl add. Add a capability probe that skips the test instead of failing when this kernel limitation is detected. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

Add `cfsctl oci run` which implements a `podman run`-like workflow with composefs integrity enforcement: pull-if-missing → verify PKCS#7 signature → mount composefs overlay → generate OCI runtime spec → exec into crun. This is the key missing piece for proving that container workloads can be run entirely through composefs-verified filesystems, with IPE `overlay_verity_validated` providing the enforcement boundary. Also adds `cfsctl oci stop` to tear down the container and unmount the composefs overlay. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

Address review feedback: - Add cleanup guard to unmount composefs overlay if spec generation or bundle write fails after mount_at succeeds; also call cleanup if exec() itself fails (only reached on error since exec replaces the process) - parse_user now warns to stderr for named users instead of silently becoming root - Add unit tests for parse_volume, parse_user, and generate_spec network modes (Host vs None namespace behaviour) - Add gid=5 to devpts mount options - Use .expect() instead of bare .unwrap() for trust_cert Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

After rebasing onto composefs-c-compat-split: - Wrap filesystem in ValidatedFileSystem before mkfs_erofs_versioned calls - Update generate_erofs_image to use mkfs_erofs_inner with FormatVersion - Remove duplicate mod tests block in cmdline.rs - Remove retain_top_level which is no longer present in generic_tree Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters force-pushed the sealing-impl branch 2 times, most recently from 1ce192a to 063ff54 Compare February 12, 2026 16:49

cgwalters commented Feb 12, 2026

View reviewed changes

cgwalters force-pushed the sealing-impl branch 3 times, most recently from 361eeb7 to 2f93e4a Compare March 6, 2026 12:24

cgwalters force-pushed the sealing-impl branch 2 times, most recently from 6b676dd to d226f55 Compare March 14, 2026 21:30

cgwalters mentioned this pull request Mar 25, 2026

WIP: Changes for booting composefs native systems coreos/fedora-coreos-config#3874

Open

cgwalters force-pushed the sealing-impl branch 2 times, most recently from 83ea13e to 13f1957 Compare April 4, 2026 12:53

cgwalters force-pushed the sealing-impl branch from 13f1957 to 40f2426 Compare April 10, 2026 20:26

cgwalters force-pushed the sealing-impl branch 2 times, most recently from 2295e33 to 0f06a47 Compare May 22, 2026 13:55

cgwalters added 15 commits May 23, 2026 08:41

fs: Add ObjectStore trait and FlatDigestStore

18df128

For compatibility with the C composefs, we need to support writing directly to a flat XX/DIGEST path, without a leading `objects/`. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>

tests: Add unit SELinux labeling tests

ceadc5e

We didn't have good converage of this before at the unit testing level. This builds on top of our prior dumpfile based test fixture. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters added 17 commits May 23, 2026 08:41

fix: Use correct Debian package name for fsverity-utils

2ad4ccc

The package was renamed from fsverity-utils to fsverity in current Debian unstable. Update the test dependency script accordingly. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

chore: Ignore docs/ directory (local planning docs)

cbf40bf

Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters force-pushed the sealing-impl branch from 0f06a47 to e0c049a Compare May 23, 2026 12:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Rough draft for updated generic OCI sealing#226

WIP: Rough draft for updated generic OCI sealing#226
cgwalters wants to merge 32 commits into
composefs:mainfrom
cgwalters:sealing-impl

cgwalters commented Feb 8, 2026

Uh oh!

cgwalters Feb 12, 2026

Uh oh!

cgwalters Feb 12, 2026

Uh oh!

cgwalters Feb 12, 2026

Uh oh!

cgwalters Feb 12, 2026

Uh oh!

cgwalters Feb 12, 2026

Uh oh!

cgwalters Feb 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgwalters commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

cgwalters commented Feb 8, 2026

Uh oh!

cgwalters Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgwalters commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant