bwrap namespace creation fails during cross-architecture install-to-filesystem

[henrywang]

When building a cross-architecture bootc image (aarch64 on x86_64 host) via bootc-image-builder, bootc install to-filesystem fails because bwrap cannot create a new namespace under QEMU user-mode emulation.

The bwrap sandbox is created by bootc (not bootupd) when invoking bootupctl backend install. Under QEMU user-mode emulation for cross-architecture builds, the clone() syscalls used by bwrap to create new namespaces return EINVAL.

org.osbuild.bootc.install-to-filesystem: 6686eb16629b6153c396e2b0e595ee6229ef20e05fbe055fd686d096e57b3bf0 {
  "kernel-args": [
    "rw",
    "console=tty0",
    "console=ttyS0"
  ],
  "target-imgref": "quay.io/bootc-test/hidden:8puv"
}
device/disk (org.osbuild.loopback): loop0 acquired (locked: False)
mount/- (org.osbuild.ext4): mounting /dev/loop0p3 -> /store/tmp/buildroot-tmp-c159qibh/mounts/
mount/boot (org.osbuild.ext4): mounting /dev/loop0p2 -> /store/tmp/buildroot-tmp-c159qibh/mounts/boot
mount/boot-efi (org.osbuild.fat): mounting /dev/loop0p1 -> /store/tmp/buildroot-tmp-c159qibh/mounts/boot/efi
Installing image: docker://quay.io/bootc-test/hidden:8puv
Initializing ostree layout
layers already present: 0; layers needed: 74 (1.1 GB)
Deploying container image...done (2 minutes)
Bootloader: grub
Installing bootloader via bootupd
bwrap: Creating new namespace failed: Invalid argument
error: Installing to filesystem: Installing bootloader: Failed to run command: Command {
    program: "bwrap",
    args: [
        "bwrap",
        "--bind",
        "/run/osbuild/mounts/ostree/deploy/default/deploy/838eff4ba7b6318e4e54925a2426e1aa0863eb2c48f189b7b5dca622fe1da446.0",
        "/",
        "--proc",
        "/proc",
        "--dev-bind",
        "/dev",
        "/dev",
        "--bind",
        "/sys",
        "/sys",
        "--tmpfs",
        "/run",
        "--bind",
        "/run",
        "/run",
        "--bind",
        "/run/osbuild/mounts/boot",
        "/boot",
        "--setenv",
        "PATH",
        "/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin",
        "--",
        "bootupctl",
        "backend",
        "install",
        "--write-uuid",
        "--device",
        "/dev/loop0",
        "/",
    ],
    create_pidfd: false,
}
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 75, in <module>
    r = main(args["options"], args["inputs"], args["paths"])
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 70, in main
    subprocess.run(pargs, env=env, check=True)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.14/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage2+/run/containers/storage:additionalimagestore=/run/osbuild/containers/storage]cb6aace90afbc418af10e1d32c060706123d53c7bac8860aa732cc3312d1cdd3', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'quay.io/bootc-test/hidden:8puv', '/run/osbuild/mounts']' returned non-zero exit status 1.
mount/boot-efi (org.osbuild.fat): umount: /store/tmp/buildroot-tmp-c159qibh/mounts/boot/efi unmounted
mount/boot (org.osbuild.ext4): umount: /store/tmp/buildroot-tmp-c159qibh/mounts/boot unmounted
mount/- (org.osbuild.ext4): umount: /store/tmp/buildroot-tmp-c159qibh/mounts/ unmounted

⏱  Duration: 446.97s
manifest - failed

Steps to reproduce

1. On an x86_64 host, use bootc-image-builder to build an aarch64 image
2. The build reaches org.osbuild.bootc.install-to-filesystem stage
3. bootc install to-filesystem tries to install the bootloader via bootupd
4. bootc wraps the bootupctl backend install call in bwrap
5. bwrap fails to create namespaces because QEMU user-mode emulation does not fully support namespace syscalls

[jbtrystram]

hmm I added this bwrap thing and did not see that one coming :)

We could turn it off when we know for sure we are running from the deployed container, e.g. no --src-imgref I guess

[cgwalters]

We could turn it off when we know for sure we are running from the deployed container, e.g. no --src-imgref I guess

Yeah, that would make sense, though it would increase our risk of skew.

---

Ultimately what I think will work best in the short term is to stop trying to use qemu-user in bib but instead always do full-system emulation, which is what we need to run images anyways.

This would need a workflow wrapper beyond just podman run though. It's basically what bootc-dev/bcvk#9 is aiming for.

[jbtrystram]

Ultimately what I think will work best in the short term is to stop trying to use qemu-user in bib but instead always do full-system emulation, which is what we need to run images anyways.

I was told that --in-vm will land soon