support MacOS

[cgwalters]

This is a key thing for us to finish replacing podman-bootc. I think a key thing here is we should act as close as possible to the Linux implementation. Ideally bcvk ephemeral works.

We can easily scrape out the kernel+initramfs from the target container image back to the host and launch krun (or via macadam) from there. The trickier bit here will be replicating the "mount target container image's root" when the files live inside the rootfs of the podman-machine VM. I am not sure if we can easily do the virtiofs thing. Maybe we could try using e.g. NFS (eww?).

[cfergeau]

9p might also be an option for the file sharing.

With newer macs (M3+) nested virtualization is supported in linux guests, I don’t know if running a nested VM inside the podman machine VM could achieve what you are after?

[cgwalters]

Also explicitly cross-referencing crc-org/macadam#140

With newer macs (M3+) nested virtualization is supported in linux guests, I don’t know if running a nested VM inside the podman machine VM could achieve what you are after?

It's a bit tricky; I think sometimes folks would want that, other times not. The biggest thing here is it scopes resource usage of the L2 VMs to the max resources of the L1 (podman machine) VM.

[gbraad]

For macadam SELinux is enabled, while for bcvk this is disabled. This is one of those things that people want to test with

[cgwalters]

For macadam SELinux is enabled, while for bcvk this is disabled.

We only emit selinux=0 for bcvk ephemeral right now, we definitely don't do that (i.e. selinux is enforcing) for the bcvk to-disk and bcvk libvirt flows.

[cgwalters]

We only emit selinux=0 for bcvk ephemeral right now,

Split that issue out to #22

[cfergeau]

The trickier bit here will be replicating the "mount target container image's root" when the files live inside the rootfs of the podman-machine VM. I am not sure if we can easily do the virtiofs thing. Maybe we could try using e.g. NFS (eww?).

Does this need to be writable? or read-only is fine? https://manpages.ubuntu.com/manpages/questing/en/man1/nbdkit-linuxdisk-plugin.1.html might work for that. vfkit can use a disk image served over nbd.

[cgwalters]

Does this need to be writable? or read-only is fine?

bcvk ephemeral always injects systemd.volatile=overlay today, so we don't depend on writes to the backing store.

https://manpages.ubuntu.com/manpages/questing/en/man1/nbdkit-linuxdisk-plugin.1.html might work for that. vfkit can use a disk image served over nbd.

To me a big goal of bcvk ephemeral is that it's O(1) to launch (i.e. it doesn't get slower as your image gets bigger). This is how podman run works because of overlayfs and general dynamic access.

It would not be O(1) to run e.g. mke2fs on a loop blockdev and serve that via nbd (and of course more generally doing so requires duplicating storage!). Now...could we somehow have a nbd server that was dynamic? In theory, maybe? I guess in particular with erofs it seems theoretically possible...

But really serving via virtiofsd or some other filesystem-based mechanism just meshes much more directly/natively with container images.

[cfergeau]

It would not be O(1) to run e.g. mke2fs on a loop blockdev and serve that via nbd

hadn’t looked at the linuxdisk plugin source code, I assumed/hoped this was dynamically generated, but now I see the mke2fs call in there :(

But really serving via virtiofsd or some other filesystem-based mechanism just meshes much more directly/natively with container images.

We don’t have that level of control over apple’s virtiofs implementation (ie vfkit), but maybe something could be done with the krunkit stack? I’m not familiar enough with it.

[cgwalters]

Did coreos/cap-std-ext#80 to help with this

[cgwalters]

We don’t have that level of control over apple’s virtiofs implementation (ie vfkit), but maybe something could be done with the krunkit stack? I’m not familiar enough with it.

cc @germag

[jtligon]

would be bcvk be an executable on the mac like bcvk run, or would it be podman machine ssh -> bcvk run?

if its the latter, then we need to rethink this concept. The solution to supporting another OS shouldn't be "lets just get them to a Linux shell so they can do linux-y things"

[cgwalters]

No the plan is the former; doing anything bcvk-like from inside a VM would add a whole massive level of complexity.

[cgwalters]

OK on this topic I did #211 - in a nutshell, supporting apple/container would probably be quite a bit easier than supporting podman machine.