docs: add sandbox design rationale (Docker vs microVMs)#1025
Conversation
Explains why Docker containers were chosen over microVMs for network sandboxing in CI/CD, covering the threat model, practical trade-offs, and defense-in-depth mitigations. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new design document explaining the rationale for using Docker containers (plus Squid + iptables) instead of microVMs for AWF’s network egress sandboxing, framing it around AWF’s threat model and operational constraints in CI.
Changes:
- Introduces
docs/sandbox-design.mdcovering threat model, Docker vs microVM trade-offs, and defense-in-depth mitigations. - Documents capability dropping, transparent interception, DNS restrictions, and the “outer VM boundary” assumption.
- Provides decision criteria for when a microVM-based approach would be more appropriate.
Comments suppressed due to low confidence (1)
docs/sandbox-design.md:138
- Same as the earlier table: rows start with
||, which creates an extra empty column in GitHub’s renderer. Adjust to standard markdown table formatting with a single leading pipe per row.
| Criterion | Docker | MicroVM |
|-----------|--------|---------|
| Sufficient for network egress control | Yes | Yes (overkill) |
| Available on GitHub Actions runners | Yes | No (needs KVM) |
| Startup overhead | ~1-2s | ~3-10s |
| Filesystem sharing | Bind mounts (fast) | virtio-fs/9p (slower) |
| Multi-container orchestration | Docker Compose | Manual networking |
| Isolation strength | Namespace (+ outer VM) | Hardware boundary |
| Operational complexity | Low | High |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | Approach | Typical startup | Notes | | ||
| |----------|----------------|-------| | ||
| | Docker container | ~1-2s | Image pull cached across runs | | ||
| | Firecracker microVM | ~3-5s | Kernel boot + rootfs mount | | ||
| | Kata Container | ~5-10s | Full VM boot with guest kernel | | ||
|
|
There was a problem hiding this comment.
The markdown table here uses an extra leading | on each row (e.g., || Approach | ...), which renders as an unintended empty first column on GitHub. Use standard GitHub markdown table syntax with a single leading pipe so the columns align correctly.
This issue also appears on line 130 of the same file.
docs/sandbox-design.md
Outdated
| volumes: | ||
| - /host-filesystem:/host:ro | ||
| - writable-home:/host/home/runner:rw |
There was a problem hiding this comment.
This compose YAML snippet appears to be presented as an example of AWF’s actual filesystem mounts, but it doesn’t match the project’s current approach (AWF defaults to selective mounting and does not mount a full host filesystem/home by default). Consider updating this section/snippet to reflect selective mounting (workspace + specific paths) and/or explicitly label it as simplified pseudo-config so readers don’t assume these are the real mounts.
| volumes: | |
| - /host-filesystem:/host:ro | |
| - writable-home:/host/home/runner:rw | |
| # Simplified example: AWF selectively mounts the workspace and explicit paths, | |
| # not the entire host filesystem or home directory. | |
| volumes: | |
| - ${GITHUB_WORKSPACE}:/workspace:rw | |
| - ~/.npm:/host/home/runner/.npm:rw | |
| - ~/.cargo:/host/home/runner/.cargo:rw |
docs/sandbox-design.md
Outdated
|
|
||
| ### Docker is pre-installed on every runner | ||
|
|
||
| Docker is available out of the box on all GitHub-hosted runner images. No setup step, no custom runner configuration, no feature flags. |
There was a problem hiding this comment.
This section says Docker is available “on all GitHub-hosted runner images,” but the repo’s compatibility docs indicate macOS and Windows runners are not supported due to Docker/Linux/iptables requirements. Suggest tightening the statement to “GitHub-hosted Ubuntu runners” or “supported Linux runners” to avoid implying cross-OS support.
| Docker is available out of the box on all GitHub-hosted runner images. No setup step, no custom runner configuration, no feature flags. | |
| Docker is available out of the box on GitHub-hosted Ubuntu runners. No setup step, no custom runner configuration, no feature flags. |
docs/sandbox-design.md
Outdated
| All HTTP (port 80) and HTTPS (port 443) traffic is redirected to Squid via iptables DNAT rules in the NAT table. This is transparent to the agent — no `HTTP_PROXY` environment variable needed, no application-level proxy configuration required. | ||
|
|
||
| :::tip | ||
| Because interception happens at the kernel level (iptables), the agent cannot bypass it from userspace without `NET_ADMIN`. |
There was a problem hiding this comment.
The text says no HTTP_PROXY environment variable is needed because interception is transparent. In AWF, HTTP_PROXY/HTTPS_PROXY are still set in the container environment, so this reads as inconsistent with the implementation. Consider rephrasing to clarify that kernel-level DNAT enforces routing even if apps ignore proxy vars, while proxy env vars are set for compatibility/defense-in-depth.
| All HTTP (port 80) and HTTPS (port 443) traffic is redirected to Squid via iptables DNAT rules in the NAT table. This is transparent to the agent — no `HTTP_PROXY` environment variable needed, no application-level proxy configuration required. | |
| :::tip | |
| Because interception happens at the kernel level (iptables), the agent cannot bypass it from userspace without `NET_ADMIN`. | |
| All HTTP (port 80) and HTTPS (port 443) traffic is redirected to Squid via iptables DNAT rules in the NAT table. This is transparent to the agent — routing is enforced even if applications ignore any `HTTP_PROXY`/`HTTPS_PROXY` environment variables; application-level proxy configuration is not required for interception to work. | |
| :::tip | |
| In AWF, `HTTP_PROXY`/`HTTPS_PROXY` may still be set in the container environment for compatibility and defense-in-depth, but kernel-level iptables DNAT enforces routing even if those variables are ignored. The agent cannot bypass this from userspace without `NET_ADMIN`. |
docs/sandbox-design.md
Outdated
|
|
||
| ## When microVMs would be the right choice | ||
|
|
||
| MicroVMs (Firecracker, Kata Containers, gVisor) provide stronger isolation at the cost of complexity and performance. They would be appropriate when: |
There was a problem hiding this comment.
gVisor isn’t a microVM runtime (it’s a userspace kernel / syscall interception sandbox). Listing it alongside Firecracker/Kata as “MicroVMs” is misleading—consider splitting this into “microVMs (Firecracker, Kata/Cloud Hypervisor)” vs “other sandboxing (gVisor)” or similar wording.
| MicroVMs (Firecracker, Kata Containers, gVisor) provide stronger isolation at the cost of complexity and performance. They would be appropriate when: | |
| MicroVMs (Firecracker, Kata Containers/Cloud Hypervisor) provide stronger isolation at the cost of complexity and performance, and other sandboxing runtimes like gVisor can also be used to harden isolation. These approaches would be appropriate when: |
docs/sandbox-design.md
Outdated
|
|
||
| - Process and memory isolation from other tenants | ||
| - Dedicated kernel and filesystem | ||
| - Network namespace separation |
There was a problem hiding this comment.
The runner VM bullet list includes “Network namespace separation,” but network namespaces are a Linux kernel feature rather than something the VM boundary itself provides. Consider rewording this bullet to “network isolation from other tenants/VMs” (or remove it) to avoid implying namespaces are provided by the runner VM.
| - Network namespace separation | |
| - Network isolation from other tenants/VMs |
|
🤖 Smoke test results for run
Overall: PASS —
|
.NET Build Test Results
Overall: PASS Run outputhello-world: json-parse:
|
Deno Build Test Results
Overall: ✅ PASS
|
Go Build Test Results
Overall: ✅ PASS
|
|
PR titles: docs: add sandbox design rationale (Docker vs microVMs); [Test Coverage] test: expand host-iptables branch coverage
|
C++ Build Test Results
Overall: PASS
|
|
Smoke Test Results ✅ GitHub MCP: #991 feat(docker): pre-seed Maven/Gradle/sbt proxy config, #992 docs: update runner and architecture compatibility Overall: PASS
|
Bun Build Test Results
Overall: PASS ✅ Bun v1.3.9
|
Build Test: Node.js ✅
Overall: PASS
|
Rust Build Test Results
Overall: PASS ✅
|
Java Build Test Results
Overall: PASS ✅ All projects compiled and tests passed successfully.
|
- Fix "network namespace separation" → "network isolation" (VM provides isolation, not Linux namespaces) - Remove standalone "Docker is pre-installed" section (macOS runners lack Docker); fold the key point into the KVM section - Remove "Composability with Docker Compose" section (weak argument) - Update volume snippet to reflect selective mounting, not full host FS - Clarify HTTP_PROXY: DNAT enforces routing regardless, proxy vars are set for compatibility/defense-in-depth - Separate gVisor (userspace kernel) from microVMs (Firecracker, Kata) - Remove "Multi-container orchestration" row from summary table Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Smoke Test ResultsLast 2 merged PRs:
Overall: PASS
|
Deno Build Test Results
Overall: ✅ PASS
|
C++ Build Test Results
Overall: PASS ✅
|
Bun Build Test Results
Overall: PASS ✅ Bun version: 1.3.9
|
Build Test: Node.js ✅
Overall: PASS
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
docs/sandbox-design.md
Outdated
| ### Capability dropping | ||
|
|
||
| The agent container starts with `NET_ADMIN` to configure iptables rules, then **drops the capability** before executing user commands: | ||
|
|
||
| ```bash | ||
| # In entrypoint.sh | ||
| exec capsh --drop=cap_net_admin -- -c "$USER_COMMAND" | ||
| ``` | ||
|
|
||
| Without `NET_ADMIN`, agent code cannot modify iptables rules to bypass the proxy. |
There was a problem hiding this comment.
The capability-dropping description and example are a bit misleading vs the actual implementation: the container is granted NET_ADMIN and SYS_CHROOT/SYS_ADMIN (src/docker-manager.ts), and entrypoint drops a mode-dependent set via CAPS_TO_DROP rather than always --drop=cap_net_admin with $USER_COMMAND. Suggest updating this section to reflect the full capability set granted during setup and that all of them are dropped before user code runs (and/or label the snippet as pseudocode).
docs/sandbox-design.md
Outdated
|
|
||
| ### DNS restriction | ||
|
|
||
| DNS traffic is restricted to whitelisted servers only (default: Google DNS `8.8.8.8`, `8.8.4.4`). This prevents DNS-based data exfiltration where an agent encodes data in DNS queries to an attacker-controlled nameserver. |
There was a problem hiding this comment.
DNS restrictions: the implementation allows DNS to the configured servers (AWF_DNS_SERVERS defaulting to 8.8.8.8/8.8.4.4) and Docker’s embedded DNS 127.0.0.11 for container name resolution (containers/agent/setup-iptables.sh). Consider mentioning 127.0.0.11 and the AWF_DNS_SERVERS override so “whitelisted servers only” matches what actually happens.
| DNS traffic is restricted to whitelisted servers only (default: Google DNS `8.8.8.8`, `8.8.4.4`). This prevents DNS-based data exfiltration where an agent encodes data in DNS queries to an attacker-controlled nameserver. | |
| DNS traffic from the sandbox is restricted to a small set of whitelisted DNS servers. By default, iptables only allows DNS to Google DNS (`8.8.8.8`, `8.8.4.4`) and to Docker’s embedded DNS at `127.0.0.11` (used for container name resolution). The allowlist can be overridden via the `AWF_DNS_SERVERS` configuration. This prevents DNS-based data exfiltration where an agent encodes data in DNS queries to an attacker-controlled nameserver. |
docs/sandbox-design.md
Outdated
| - ~/.npm:/host/home/runner/.npm:rw | ||
| - ~/.cargo:/host/home/runner/.cargo:rw |
There was a problem hiding this comment.
The bind-mount example hard-codes the GitHub-hosted Linux home path (/home/runner) in the container destination. Since the code uses the effective $HOME (and this doc may be read in non-GitHub-hosted contexts), consider using ${HOME}/${effectiveHome}-style placeholders in the example to avoid implying the path is always /home/runner.
| - ~/.npm:/host/home/runner/.npm:rw | |
| - ~/.cargo:/host/home/runner/.cargo:rw | |
| - ${HOME}/.npm:/host${HOME}/.npm:rw | |
| - ${HOME}/.cargo:/host${HOME}/.cargo:rw |
docs/sandbox-design.md
Outdated
| ### GitHub Actions runners are already VMs | ||
|
|
||
| Each GitHub Actions runner is an isolated virtual machine. The runner VM provides: | ||
|
|
||
| - Process and memory isolation from other tenants | ||
| - Dedicated kernel and filesystem | ||
| - Network isolation from other tenants/VMs | ||
|
|
||
| Adding a microVM inside this VM would create **nested virtualization** — a VM inside a VM — with minimal additional security benefit for network-only filtering. | ||
|
|
There was a problem hiding this comment.
The section title/wording says “GitHub Actions runners are already VMs”, but later you specifically rely on GitHub-hosted runner constraints (e.g., /dev/kvm not exposed). Consider tightening this to “GitHub-hosted Actions runners…” (and optionally calling out self-hosted runners may be bare metal/containers) to keep the doc internally consistent and avoid overgeneralizing.
| ### GitHub Actions runners are already VMs | |
| Each GitHub Actions runner is an isolated virtual machine. The runner VM provides: | |
| - Process and memory isolation from other tenants | |
| - Dedicated kernel and filesystem | |
| - Network isolation from other tenants/VMs | |
| Adding a microVM inside this VM would create **nested virtualization** — a VM inside a VM — with minimal additional security benefit for network-only filtering. | |
| ### GitHub-hosted Actions runners are already VMs | |
| Each GitHub-hosted GitHub Actions runner is an isolated virtual machine (VM). The runner VM provides: | |
| - Process and memory isolation from other tenants | |
| - Dedicated kernel and filesystem | |
| - Network isolation from other tenants/VMs | |
| Self-hosted runners, by contrast, may run on bare metal, in containers, or in other VM environments; this document focuses on GitHub-hosted runners. | |
| Adding a microVM inside this VM on a GitHub-hosted runner would create **nested virtualization** — a VM inside a VM — with minimal additional security benefit for network-only filtering. |
docs/sandbox-design.md
Outdated
|
|
||
| | Approach | Typical startup | Notes | | ||
| |----------|----------------|-------| | ||
| | Docker container | ~1-2s | Image pull cached across runs | |
There was a problem hiding this comment.
The “Image pull cached across runs” note is not generally true on GitHub-hosted runners (jobs run on fresh ephemeral VMs, so cache persistence across runs isn’t guaranteed). Suggest rephrasing to something like “often pre-cached on the runner image / may be cached within a job” to avoid implying reliable cross-run caching.
| | Docker container | ~1-2s | Image pull cached across runs | | |
| | Docker container | ~1-2s | Base image often pre-cached on GitHub runner; pulls may be cached within a job | |
docs/sandbox-design.md
Outdated
| |----------|----------------|-------| | ||
| | Docker container | ~1-2s | Image pull cached across runs | | ||
| | Firecracker microVM | ~3-5s | Kernel boot + rootfs mount | | ||
| | Kata Container | ~5-10s | Full VM boot with guest kernel | |
There was a problem hiding this comment.
Spelling/naming: “Kata Container” should be “Kata Containers” (matches earlier usage and the project name).
| | Kata Container | ~5-10s | Full VM boot with guest kernel | | |
| | Kata Containers | ~5-10s | Full VM boot with guest kernel | |
Go Build Test Results ✅
Overall: PASS
|
Rust Build Test Results
Overall: ✅ PASS
|
Java Build Test Results
Overall: ✅ PASS
|
|
GitHub MCP: ✅ — feat(docker): pre-seed Maven/Gradle/sbt proxy config in agent container; docs: update runner and architecture compatibility
|
.NET Build Test Results
Overall: PASS Run outputhello-world: json-parse:
|
|
Smoke Test Results — ✅ GitHub MCP: Overall: PASS
|
- Specify "GitHub-hosted" runners, note self-hosted may differ
- Fix Docker caching claim (ephemeral VMs don't guarantee cross-run cache)
- Fix "Kata Container" → "Kata Containers" spelling
- Use ${HOME} placeholders instead of hardcoded /home/runner paths
- Document full capability set (NET_ADMIN, SYS_CHROOT, SYS_ADMIN) and
label code snippet as pseudocode
- Add Docker embedded DNS (127.0.0.11) and --dns-servers override to
DNS restriction section
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Build Test: Node.js Results
Overall: PASS ✅
|
🦕 Deno Build Test Results
Overall: ✅ PASS
|
|
🔬 Smoke test results for run ✅ GitHub MCP — Last 2 merged PRs: Overall: PASS | Author:
|
C++ Build Test Results
Overall: PASS
|
🦀 Rust Build Test Results
Overall: ✅ PASS
|
.NET Build Test Results
Overall: PASS Run outputhello-world: json-parse:
|
Go Build Test Results
Overall: ✅ PASS
|
🧪 Build Test: Bun
Overall: ✅ PASS
|
Smoke Test Results✅ GitHub MCP — PR #991: "feat(docker): pre-seed Maven/Gradle/sbt proxy config in agent container" | PR #963: "fix: set JAVA_TOOL_OPTIONS and generate Maven settings.xml for JVM proxy" Overall: PASS
|
Java Build Test Results
Overall: PASS ✅
|
|
Merged PRs: feat(docker): pre-seed Maven/Gradle/sbt proxy config in agent container | docs: update runner and architecture compatibility
|
1 participant
Summary
docs/sandbox-design.mdexplaining why Docker containers were chosen over microVMs for network sandboxingTest plan
🤖 Generated with Claude Code