Feature or enhancement request details
Summary
Propose that apple/containerization adopt a public, cross-runtime performance benchmark suite as part of its regression and release validation process - measuring volume I/O, container-to-container networking, startup latency, port-mapped TTFB, small-file throughput, and image build performance against the runtimes Apple Silicon developers use day-to-day (Colima, Docker Desktop, OrbStack).
@zot24 has published a reproducible suite covering these dimensions, with runs on both Sequoia (15.7) and Tahoe (26.4):
Why this matters for apple/containerization
Apple Container is the only runtime in the comparison built on a 1-VM-per-container isolation model. That's an architectural choice with real upside - full kernel-level boundary per workload, strongest isolation in this comparison - but it also means the project sits on a different curve than the shared-VM runtimes, and developers picking a runtime for daily use will compare them directly anyway.
Public, repeatable performance numbers tracked release-over-release would:
- Make it visible when an OS-level change (Virtualization.framework, vmnet, kernel I/O paths) moves the needle for Apple Container.
- Give the team early-warning regression signal before users notice.
- Document wins (e.g., volume I/O is already competitive on writes) so the picture isn't dominated by the axes that need the most work.
- Establish Apple as the runtime that publishes its own numbers - which, given the hardware advantage on Apple Silicon, is a strong long-term position.
Snapshot of current measurements (Tahoe 26.4.1, M3, 16GB)
From @zot24's published results - 4 runs × 3 iterations per metric, per-container limits enforced at --cpus 4 --memory 4g:
| Test |
Colima |
Docker Desktop |
OrbStack |
Apple Container |
| Container Startup |
0.291s |
0.329s |
0.371s |
0.935s |
| Volume Write (256MB seq.) |
1,306 MB/s |
1,327 MB/s |
1,566 MB/s |
1,280 MB/s |
| Volume Read (256MB seq.) |
3,354 MB/s |
2,970 MB/s |
10,061 MB/s |
2,765 MB/s |
| Small Files (1000 × 4KB) |
0.761s |
0.711s |
0.620s |
1.257s |
| HTTP Fetch (curl) |
0.830s |
0.716s |
0.774s |
0.771s |
| C2C Throughput (iperf3) |
110.3 Gbps |
119.1 Gbps |
130.2 Gbps |
23.0 Gbps |
| Port-mapped TTFB (nginx) |
1.56ms |
2.69ms |
2.18ms |
3.24ms |
| Image Build (Dockerfile) |
6.81s |
4.98s |
5.28s |
n/a* |
* Build skipped because the builder VM has no outbound network access, so apk add (and any other package installer) fails.
What's strong
- Volume write throughput (1,280 MB/s) is within 4% of Docker Desktop and 2% of Colima - Apple Container is in the same class as the shared-VM runtimes on sequential write.
- HTTP fetch (0.771s) is second-fastest in the set - tied with OrbStack (0.774s) and ahead of Colima (0.830s).
- The 1-VM-per-container isolation model has no peer in the comparison - the right architectural foundation for security-sensitive and multi-tenant workloads, and worth keeping prominent in messaging.
- Tahoe-era Virtualization.framework gains lifted volume-write numbers across the three runtimes that ran on both releases (+61% to +104%). The underlying platform work is paying off across the ecosystem.
Where the gaps are, and likely root causes
Presented as observations from the benchmark plus a working hypothesis for each, not as criticism - the goal is to make them measurable so progress is visible.
-
Container startup: 0.935s vs 0.291s (Colima) - ~3.2x slower.
Hypothesis: full VM kernel boot per container, vs. namespace creation inside an already-running VM. The architectural choice is deliberate, but the absolute number is something users feel on every container run and every test invocation. Worth tracking whether kernel image size, paravirt boot path, or VM warm-pooling could close some of that gap without compromising the isolation boundary.
-
C2C throughput: 23.0 Gbps vs 130.2 Gbps (OrbStack) - ~5.7x slower.
Per the benchmark notes, this is attributed to per-container vmnet networking; the shared-VM runtimes have an in-VM bridge between containers. This is the largest single gap and probably the biggest pain point for multi-container workloads (databases + app servers, microservice fan-out, build pipelines). Would benefit from a public roadmap item: shared overlay, vsock-based fast path, or accelerated inter-VM transport.
-
Image build: blocked entirely - the builder VM cannot reach the internet, so any Dockerfile that pulls dependencies (apk add, apt-get, pip install, npm install, etc.) fails. This is the single biggest blocker to daily-driver adoption today, because almost every real-world Dockerfile pulls something. Even an opt-in network mode for the builder would unblock most users.
-
Small-file write: 1.257s vs 0.620s (OrbStack) - ~2x slower. Likely a shared-volume metadata path effect; relevant for node_modules, Python venvs, and Git operations inside containers.
-
Port-mapped TTFB: 3.24ms vs 1.56ms (Colima) - ~2x slower. Relevant for local web development workflows where the user is hitting the container from the host browser repeatedly.
The actual request
Incorporate a benchmark suite of this shape into the project's regression and release process. Concretely:
-
Adopt or fork the existing suite. @zot24's benchmark.sh + aggregate.py + charts.py is MIT-licensed and runs all four runtimes on the same machine in the same session, with outlier filtering and per-container limits. Forking it (with attribution) gives the project a turnkey baseline.
-
Track the eight metrics above as release gates / dashboards. Publish per-release numbers under apple/containerization so users see progress, and so regressions surface immediately rather than being discovered by users in the field.
-
Run the suite cross-OS (e.g., macOS 26.x point releases, future major versions). The Sequoia → Tahoe deltas in @zot24's data show how much OS-level changes matter; an Apple-side baseline means Apple sees its own platform's effect on its own runtime first.
-
Publish the methodology and raw results. Even if Apple's internal benchmarks are more comprehensive, a public reproducible subset gives the community a shared reference and removes ambiguity in third-party comparisons. Publishing the project's own numbers establishes leadership; otherwise the framing gets set by whoever else publishes first.
-
Optionally extend the suite with metrics specific to Apple Container's strengths - VM-boundary isolation tests, attack-surface measurements, cold-start-from-snapshot if/when implemented - so the comparison highlights where the architectural choice pays off.
What this looks like adopted
- A
benchmarks/ directory in the repo with a runnable script and documented setup.
- A CI job (or scheduled workflow on representative hardware) that produces a results JSON per release.
- A
BENCHMARKS.md that publishes the latest numbers and the trend over the last N releases.
- Cross-runtime numbers kept honest - including the axes where competitors are ahead - because that's how regression suites earn trust and how the team gets the clearest signal about where to invest.
References
Open to helping with metric definition or contributing to a fork of the suite.
Code of Conduct
Feature or enhancement request details
Summary
Propose that
apple/containerizationadopt a public, cross-runtime performance benchmark suite as part of its regression and release validation process - measuring volume I/O, container-to-container networking, startup latency, port-mapped TTFB, small-file throughput, and image build performance against the runtimes Apple Silicon developers use day-to-day (Colima, Docker Desktop, OrbStack).@zot24 has published a reproducible suite covering these dimensions, with runs on both Sequoia (15.7) and Tahoe (26.4):
Why this matters for
apple/containerizationApple Container is the only runtime in the comparison built on a 1-VM-per-container isolation model. That's an architectural choice with real upside - full kernel-level boundary per workload, strongest isolation in this comparison - but it also means the project sits on a different curve than the shared-VM runtimes, and developers picking a runtime for daily use will compare them directly anyway.
Public, repeatable performance numbers tracked release-over-release would:
Snapshot of current measurements (Tahoe 26.4.1, M3, 16GB)
From @zot24's published results - 4 runs × 3 iterations per metric, per-container limits enforced at
--cpus 4 --memory 4g:* Build skipped because the builder VM has no outbound network access, so
apk add(and any other package installer) fails.What's strong
Where the gaps are, and likely root causes
Presented as observations from the benchmark plus a working hypothesis for each, not as criticism - the goal is to make them measurable so progress is visible.
Container startup: 0.935s vs 0.291s (Colima) - ~3.2x slower.
Hypothesis: full VM kernel boot per container, vs. namespace creation inside an already-running VM. The architectural choice is deliberate, but the absolute number is something users feel on every
container runand every test invocation. Worth tracking whether kernel image size, paravirt boot path, or VM warm-pooling could close some of that gap without compromising the isolation boundary.C2C throughput: 23.0 Gbps vs 130.2 Gbps (OrbStack) - ~5.7x slower.
Per the benchmark notes, this is attributed to per-container vmnet networking; the shared-VM runtimes have an in-VM bridge between containers. This is the largest single gap and probably the biggest pain point for multi-container workloads (databases + app servers, microservice fan-out, build pipelines). Would benefit from a public roadmap item: shared overlay, vsock-based fast path, or accelerated inter-VM transport.
Image build: blocked entirely - the builder VM cannot reach the internet, so any Dockerfile that pulls dependencies (
apk add,apt-get,pip install,npm install, etc.) fails. This is the single biggest blocker to daily-driver adoption today, because almost every real-world Dockerfile pulls something. Even an opt-in network mode for the builder would unblock most users.Small-file write: 1.257s vs 0.620s (OrbStack) - ~2x slower. Likely a shared-volume metadata path effect; relevant for
node_modules, Python venvs, and Git operations inside containers.Port-mapped TTFB: 3.24ms vs 1.56ms (Colima) - ~2x slower. Relevant for local web development workflows where the user is hitting the container from the host browser repeatedly.
The actual request
Incorporate a benchmark suite of this shape into the project's regression and release process. Concretely:
Adopt or fork the existing suite. @zot24's
benchmark.sh+aggregate.py+charts.pyis MIT-licensed and runs all four runtimes on the same machine in the same session, with outlier filtering and per-container limits. Forking it (with attribution) gives the project a turnkey baseline.Track the eight metrics above as release gates / dashboards. Publish per-release numbers under
apple/containerizationso users see progress, and so regressions surface immediately rather than being discovered by users in the field.Run the suite cross-OS (e.g., macOS 26.x point releases, future major versions). The Sequoia → Tahoe deltas in @zot24's data show how much OS-level changes matter; an Apple-side baseline means Apple sees its own platform's effect on its own runtime first.
Publish the methodology and raw results. Even if Apple's internal benchmarks are more comprehensive, a public reproducible subset gives the community a shared reference and removes ambiguity in third-party comparisons. Publishing the project's own numbers establishes leadership; otherwise the framing gets set by whoever else publishes first.
Optionally extend the suite with metrics specific to Apple Container's strengths - VM-boundary isolation tests, attack-surface measurements, cold-start-from-snapshot if/when implemented - so the comparison highlights where the architectural choice pays off.
What this looks like adopted
benchmarks/directory in the repo with a runnable script and documented setup.BENCHMARKS.mdthat publishes the latest numbers and the trend over the last N releases.References
charts/tahoe-comparison.png,charts/sequoia-vs-tahoe.png,charts/delta-heatmap.pngOpen to helping with metric definition or contributing to a fork of the suite.
Code of Conduct