Skip to content

fix(sandbox): resolve symlinked binary paths in network policy matching#774

Open
johntmyers wants to merge 4 commits intomainfrom
fix/770-symlink-binary-resolution
Open

fix(sandbox): resolve symlinked binary paths in network policy matching#774
johntmyers wants to merge 4 commits intomainfrom
fix/770-symlink-binary-resolution

Conversation

@johntmyers
Copy link
Copy Markdown
Collaborator

@johntmyers johntmyers commented Apr 6, 2026

Summary

Policy binary paths specified as symlinks (e.g., /usr/bin/python3) were silently denied because the kernel reports the canonical path via /proc/<pid>/exe (e.g., /usr/bin/python3.11). This fix resolves symlinks through the container filesystem after the entrypoint starts, expanding the OPA policy data so both the original and resolved paths match.

Related Issue

Closes #770

Changes

  • opa.rs: Added resolve_binary_in_container() helper that resolves symlinks via /proc/<pid>/root/ on Linux using iterative read_link (not canonicalize, which resolves the procfs mount itself). Added from_proto_with_pid() and reload_from_proto_with_pid() methods that expand binary paths during OPA data construction. Existing from_proto() / reload_from_proto() delegate with pid=0 (backward-compatible, no expansion). Added normalize_path() for relative symlink targets with .. components.
  • lib.rs: load_policy() now retains the proto for post-start OPA rebuild. After entrypoint_pid.store(), triggers a one-shot OPA rebuild with the real PID. run_policy_poll_loop() passes the PID on each hot-reload so symlinks are re-resolved.
  • sandbox-policy.rego: Deny reason for binary mismatches now leads with SYMLINK HINT and includes actionable fix guidance (readlink -f command, what to check in logs).

Design decisions

  • Expand policy data, not evaluation logic — the Rego rules and per-request evaluation path are untouched. Only the OPA data (binary list) is enriched at load time. This avoids introducing new code in the security-critical hot path.
  • Graceful degradation — if symlink resolution fails for any reason, the original path is preserved and behavior is identical to before this change. Resolution is best-effort.
  • No Rego changes needed — the existing b.path == exec.path strict equality naturally matches the expanded entry.
  • read_link over canonicalizestd::fs::canonicalize resolves /proc/<pid>/root itself (a kernel pseudo-symlink to /), stripping the prefix needed for path extraction. We use iterative read_link which reads only the specified symlink target, staying within the container namespace.

Best-effort approach and known risks

Symlink resolution is opportunistic — it improves the common case but cannot be guaranteed in all environments. When resolution fails, we are loud about it: per-binary WARN-level logs explain exactly what failed and what the operator should do. Deny reasons include prominent SYMLINK HINT text with actionable fix commands. Both flow through the gRPC LogPushLayer and are visible via openshell logs.

Environments where resolution will not work:

Environment Reason User impact
Restricted ptrace scope (kernel.yama.ptrace_scope >= 2) /proc/<pid>/root/ returns EACCES even for own PID Symlinks must be specified as canonical paths in policy
Rootless containers (rootless Docker, Podman) User namespace isolation prevents procfs root traversal Same — canonical paths required
Kubernetes pods without elevated security context Default seccomp/AppArmor profiles may block procfs root access Same — canonical paths required
Standalone/local mode (--policy-rules/--policy-data, no --sandbox-id) No retained proto to rebuild, no gRPC log push Resolution doesn't run; deny reasons appear on stdout only
Multi-level symlinks through /etc/alternatives Should work (iterative loop handles chains up to 40 levels), but unusual layouts may produce unexpected resolved paths Verify with readlink -f inside sandbox
Dynamically created symlinks after container start Resolution runs at startup and on policy reload, not continuously New symlinks won't be resolved until next policy reload

In all failure cases: the original user-specified path is preserved, the deny behavior is identical to pre-fix, and the operator gets a clear warning log explaining why resolution didn't work and what to do about it.

Testing

  • mise run pre-commit passes
  • 19 new unit tests covering:
    • normalize_path helper for ../. resolution
    • resolve_binary_in_container edge cases (glob skip, pid=0, nonexistent paths)
    • Expanded binary matching (resolved path allowed, original preserved, unrelated binaries denied)
    • Ancestor matching with expanded paths
    • Proto round-trips with _with_pid variants
    • Hot-reload behavior (engine replacement, symlink expansion on reload, LKG preservation)
    • Deny reason includes SYMLINK HINT and readlink -f command
    • Linux-specific e2e tests with real symlinks (single-level, multi-level, non-symlink, full proto-to-decision, hot-reload before/after) — gracefully skip in restricted environments
  • All 452 existing + new tests pass (449 sandbox + 5 integration)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Policy binary paths specified as symlinks (e.g., /usr/bin/python3) were
silently denied because the kernel reports the canonical path via
/proc/<pid>/exe (e.g., /usr/bin/python3.11). The strict string equality
in Rego never matched.

Expand policy binary paths by resolving symlinks through the container
filesystem (/proc/<pid>/root/) after the entrypoint starts. The OPA data
now contains both the original and resolved paths, so Rego's existing
strict equality check naturally matches either.

- Add resolve_binary_in_container() helper for Linux symlink resolution
- Add from_proto_with_pid() and reload_from_proto_with_pid() to OpaEngine
- Trigger one-shot OPA rebuild after entrypoint_pid is stored
- Thread entrypoint_pid through run_policy_poll_loop for hot-reloads
- Improve deny reason with symlink debugging hint
- Add 18 new tests including hot-reload and Linux symlink e2e tests

Closes #770
@johntmyers johntmyers requested a review from a team as a code owner April 6, 2026 22:44
@johntmyers johntmyers self-assigned this Apr 6, 2026
@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Apr 6, 2026
…naccessible

The Linux-specific symlink resolution tests depend on /proc/<pid>/root/
being readable, which requires CAP_SYS_PTRACE or permissive ptrace
scope. This is unavailable in CI containers, rootless containers, and
hardened hosts. Add a procfs_root_accessible() guard that skips these
tests gracefully instead of failing.
…improve deny messages

When /proc/<pid>/root/ is inaccessible (restricted ptrace, rootless
containers, hardened hosts), resolve_binary_in_container now logs a
per-binary warning with the specific error, the path it tried, and
actionable guidance (use canonical path or grant CAP_SYS_PTRACE).
Previously this was completely silent.

The Rego deny reason for binary mismatches now leads with 'SYMLINK HINT'
and includes a concrete fix command ('readlink -f' inside the sandbox)
plus what to look for in logs if automatic resolution isn't working.
…ution

std::fs::canonicalize resolves /proc/<pid>/root itself (a kernel
pseudo-symlink to /) which strips the prefix needed for path extraction.
This caused resolution to silently fail in all environments, not just CI.

Replace with an iterative read_link loop that walks the symlink chain
within the container namespace without resolving the /proc mount point.
Add normalize_path helper for relative symlink targets containing ..
components. Update procfs_root_accessible test guard to actually probe
the full resolution path instead of just checking path existence.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Sandbox egress proxy checks resolved binary path, not symlink — python3 silently blocked

1 participant