Skip to content

Add vcsim support for container backed VMs able to run nested containers#3976

Draft
hickeng wants to merge 5 commits intovmware:mainfrom
hickeng:vcsim-nested-containers
Draft

Add vcsim support for container backed VMs able to run nested containers#3976
hickeng wants to merge 5 commits intovmware:mainfrom
hickeng:vcsim-nested-containers

Conversation

@hickeng
Copy link
Copy Markdown
Contributor

@hickeng hickeng commented Mar 6, 2026

Add RUN.nestedContainers ExtraConfig option to enable running Kubernetes
or other container workloads inside container-backed VMs. When enabled,
the container is created with kind-style flags:

  • --cgroupns=private (instead of host) for proper cgroup v2 delegation
  • --security-opt seccomp=unconfined for systemd/containerd syscalls
  • --security-opt apparmor=unconfined to disable AppArmor restrictions
  • --tmpfs /tmp,/run for systemd compatibility
  • --volume /var for persistent containerd/kubelet data
  • --volume /lib/modules:/lib/modules:ro for iptables/networking
  • --device /dev/fuse for fuse-overlayfs snapshotter

In support of the above, add PropertyDiff and Checkpoint functions to generate PropertyChange
arrays by comparing two states of a managed object. This enables granular property change
notifications when modifying simulator objects.
This was necessitated by waiting for IP assignment, and Summary.Guest.IpAddress and similar
nested fields were not generating PropertyChange entries.

The typical usage pattern is:

  checkpoint := Checkpoint(vm)
  // ... make changes to vm ...
  changes := PropertyDiff(checkpoint, vm)
  ctx.Update(vm, changes)

hickeng and others added 4 commits April 15, 2026 13:55
- Add commandError helper for consistent error logging with stderr
- Add RUN.network ExtraConfig option for container network selection
  (needed for rootless podman to get IP assignment via bridge network)
- Add RUN.mountdmi error hint for rootless podman permission issues
- Update syncNetworkConfigToVMGuestProperties to trigger property change
  notifications via ctx.Update for all modified properties
- Retry IP sync after container start to handle delayed IP assignment
- Update watchContainer callback signature to include Context

These changes improve vcsim's compatibility with podman-docker and
enable proper WaitForIP functionality for container-backed VMs.

Made-with: Cursor
Signed-off-by: George Hicken <george.hicken@broadcom.com>
…king

Add PropertyDiff and Checkpoint functions to generate PropertyChange arrays
by comparing two states of a managed object. This enables granular property
change notifications when modifying simulator objects.

The typical usage pattern is:
  checkpoint := Checkpoint(vm)
  // ... make changes to vm ...
  changes := PropertyDiff(checkpoint, vm)
  ctx.Update(vm, changes)

PropertyDiff correctly handles:
- Simple field changes (Assign operation)
- Nil to value transitions (Add operation)
- Value to nil transitions (Remove operation)
- Nested struct fields
- Embedded/anonymous fields
- Slice fields

Tests include:
- Whitebox unit tests for PropertyDiff, Checkpoint, and determineChangeOp
- Simulator integration tests verifying changes are visible via PropertyCollector
- Container-backed VM test verifying network property changes from container inspect

Made-with: Cursor
Signed-off-by: George Hicken <george.hicken@broadcom.com>
…e Guest.Net

Refactor syncNetworkConfigToVMGuestProperties to use PropertyDiff for
generating granular property changes instead of manually constructing
PropertyChange arrays. This simplifies the code and ensures all modified
properties trigger proper notifications.

Key changes:
- Use Checkpoint/PropertyDiff pattern to track all VM state changes
- Populate Guest.Net from all container networks (not just when pre-configured)
- Include detailed NIC info: Network name, MacAddress, IpAddress, IpConfig
- Populate Guest.IpStack with DNS config and default route
- Update Summary.Guest.HostName in addition to IpAddress

The test now verifies:
- Guest.Net is empty before power on
- Guest.Net is populated with detailed NIC info after power on
- Each NIC has Network, MacAddress, IpAddress, Connected, and IpConfig
- Guest.IpStack contains DNS and routing information
- Summary.Guest includes both IpAddress and HostName

Made-with: Cursor
Signed-off-by: George Hicken <george.hicken@broadcom.com>
Add RUN.nestedContainers ExtraConfig option to enable running Kubernetes
or other container workloads inside container-backed VMs. When enabled,
the container is created with kind-style flags:

- --cgroupns=private (instead of host) for proper cgroup v2 delegation
- --security-opt seccomp=unconfined for systemd/containerd syscalls
- --security-opt apparmor=unconfined to disable AppArmor restrictions
- --tmpfs /tmp,/run for systemd compatibility
- --volume /var for persistent containerd/kubelet data
- --volume /lib/modules:/lib/modules:ro for iptables/networking
- --device /dev/fuse for fuse-overlayfs snapshotter

Also adds comprehensive documentation for all RUN.* ExtraConfig options
in container_virtual_machine.go with usage examples.

Reference: https://github.com/kubernetes-sigs/kind/blob/main/pkg/cluster/internal/providers/docker/provision.go

AI-Tool-Used: Cursor
AI-Tool-Use-Level: high
AI-Code-Category: non-production
Co-authored-by: Claude Opus 4 <noreply@anthropic.com>
Made-with: Cursor
Signed-off-by: George Hicken <george.hicken@broadcom.com>
@hickeng hickeng force-pushed the vcsim-nested-containers branch 2 times, most recently from 52c4c25 to 2c0856a Compare April 15, 2026 21:07
Also update tests to use require/assert instead of if/else with t.Error/Fatal for conciseness.

Signed-off-by: George Hicken <george.hicken@broadcom.com>
@hickeng hickeng force-pushed the vcsim-nested-containers branch from 2c0856a to fb87509 Compare April 15, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant