Add a dedicated Chutes sidecar route without changing existing /v1/chat/completions behavior

[mondaylord]

Summary

We want to integrate Chutes models into vllm-proxy while preserving the current production-stable behavior for existing users.

Instead of modifying the current /v1/chat/completions path, add a dedicated Chutes route:

• POST /v1/chutes/chat/completions
• GET /v1/chutes/models

This lets upstream gateways (e.g., Redpill) continue receiving standard OpenAI-style /v1/chat/completions requests and selectively forward Chutes-bound traffic to the new sidecar route.

Goals

• Keep existing /v1/chat/completions behavior unchanged
• Minimize coupling/risk to current production path
• Add Chutes integration behind explicit route + config flag

Proposed behavior

Existing route (unchanged)

POST /v1/chat/completions

• continues to forward to local vLLM backend as today

New Chutes route

POST /v1/chutes/chat/completions

• reuses existing E2EE parse/decrypt logic on ingress
• forwards plaintext request to Chutes OpenAI-compatible endpoint over TLS:
  • ${CHUTES_BASE_URL}/v1/chat/completions
  • with Authorization: Bearer ${CHUTES_API_KEY}
• reuses existing E2EE encrypt logic on egress

New Chutes models route

GET /v1/chutes/models

• proxies ${CHUTES_BASE_URL}/v1/models with Chutes auth header

Config

• CHUTES_ENABLED (default false)
• CHUTES_BASE_URL (default https://llm.chutes.ai)
• CHUTES_API_KEY (required if enabled)

Security note

This is not pure client-to-model cryptographic E2EE.
It is a practical TEE-mediated segmented model:

• Client ↔ Proxy: existing E2EE
• Proxy ↔ Chutes: TLS
• Plaintext is only visible inside trusted runtime boundaries

Acceptance criteria

1. Existing route behavior remains unchanged
2. Chutes route supports stream + non-stream
3. Existing E2EE nonce/replay checks remain in effect
4. Misconfig returns explicit 503 errors
5. No API-key leakage in logs

[mondaylord]

Implementation proposal (minimal-risk, side-by-side):

1. Keep existing route unchanged

• Do not alter current behavior of POST /v1/chat/completions
• Existing local vLLM path remains default and stable

2. Add Chutes sidecar endpoints

• POST /v1/chutes/chat/completions
• GET /v1/chutes/models

3. Reuse current E2EE logic

• Parse/decrypt ingress using existing parse_e2ee_context + decrypt_request_json
• Keep replay checks via existing nonce/timestamp claim path
• Encrypt egress using existing chunk/response encrypt helpers

4. Forwarding behavior

• Existing route -> local vLLM backend URL (unchanged)
• New Chutes route -> ${CHUTES_BASE_URL}/v1/chat/completions via TLS
• Add outbound Authorization: Bearer ${CHUTES_API_KEY} only on Chutes path

5. Config and guardrails

• CHUTES_ENABLED (feature flag)
• CHUTES_BASE_URL
• CHUTES_API_KEY
• If disabled/missing key, return explicit 503 errors (chutes_disabled / chutes_misconfigured)

6. Routing from upstream gateway

• Upstream gateway (e.g. Redpill) continues receiving standard /v1/chat/completions
• It can route Chutes-intended traffic to /v1/chutes/chat/completions by model prefix or tenant policy

7. Security framing

• This is a practical segmented model:
  • Client <-> Proxy: existing E2EE
  • Proxy <-> Chutes: TLS
• Not pure client-to-model cryptographic E2EE, but preserves current client contract and minimizes migration risk

[mondaylord]

Proposing a new endpoint: GET /v1/attestation/chain to provide end-to-end trust for the proxy->Chutes path.

Why we need this

When Redpill routes model traffic through our vllm-proxy (CVM), users currently have two separate trust concerns:

1. Is the proxy CVM genuine and running expected code?
2. Is the upstream Chutes GPU TEE genuine for the target model?

/v1/attestation/report can prove one side at a time, but it does not cryptographically bind both proofs into a single verifiable statement for one request context.

Goal

Provide a verifiable chain that lets clients check:

• proxy attestation is valid,
• Chutes attestation is valid,
• and the proxy TEE explicitly attests that it fetched/used that exact Chutes attestation (not a stale/unrelated one).

Proposed API

GET /v1/attestation/chain

Suggested query params:

• model (required): upstream model id used for Chutes attestation
• nonce (required): client challenge to prevent replay
• signing_algo (optional, default ecdsa)

Suggested response shape:

{
  "version": "1",
  "proxy": {
    "attestation": {"...": "existing /v1/attestation/report payload"},
    "signing_public_key": "..."
  },
  "upstream": {
    "provider": "chutes",
    "base_url": "https://api.chutes.ai",
    "model": "moonshotai/Kimi-K2.5-TEE",
    "attestation": {"...": "raw chutes attestation"},
    "attestation_sha256": "..."
  },
  "binding_proof": {
    "payload": {
      "nonce": "...",
      "timestamp": 1775032500,
      "provider": "chutes",
      "upstream_base_url": "https://api.chutes.ai",
      "model": "moonshotai/Kimi-K2.5-TEE",
      "upstream_attestation_sha256": "..."
    },
    "signature": "0x...",
    "signing_algo": "ecdsa",
    "signing_address": "0x..."
  }
}

Verification model (client side)

Client verifies in this order:

1. Verify proxy.attestation as genuine CVM evidence.
2. Verify upstream.attestation using Chutes verification rules.
3. Recompute sha256(upstream.attestation_raw) and compare with upstream.attestation_sha256 and binding_proof.payload.upstream_attestation_sha256.
4. Verify binding_proof.signature against proxy signing identity from attestation.
5. Ensure binding_proof.payload.nonce equals the caller-provided nonce and timestamp is fresh.

If all checks pass, we get cryptographic linkage:

this verified proxy TEE signed a statement about this exact Chutes attestation for this nonce/model.

Security properties

• Replay resistance: caller nonce + freshness window.
• Context binding: proof binds model + upstream base URL + attestation hash.
• Composability: existing /v1/attestation/report remains unchanged; /v1/attestation/chain is additive.

Backward compatibility

• Keep current /v1/attestation/report behavior.
• Add /v1/attestation/chain as opt-in for stronger verification workflows.

Implementation plan

1. Add new route and response schema.
2. Fetch proxy attestation internally (reuse existing path/service logic).
3. Fetch Chutes attestation with same nonce and requested model.
4. Canonicalize and hash upstream attestation JSON (stable serialization).
5. Build binding_proof.payload and sign with existing proxy signing key.
6. Add tests:
   • happy path,
   • nonce missing/too short,
   • stale timestamp,
   • upstream attestation fetch failure,
   • hash/signature mismatch negative tests.

Notes

This endpoint does not claim proxy and Chutes are the same TEE. It proves a stronger and practical property: a verified proxy TEE has produced a signed statement binding a specific upstream Chutes attestation to a caller challenge.

[mondaylord]

Implemented proposal update for Chutes verification architecture in vllm-proxy.

Problem

Current flow requires clients to verify both vllm-proxy and Chutes attestations (+ binding proof), which is accurate but heavier for integrators.

Proposed dual-mode design

Add verify_mode to /v1/attestation/chain:

• proxy (default): proxy verifies Chutes evidence internally and returns proxy attestation + signed verification receipt.
• passthrough: return full chain materials (proxy + chutes + binding proof) for clients who run private-ai-verifier themselves.

Verification checks used by proxy mode (aligned with private-ai-verifier chutes.py)

1. TDX status must be UpToDate.
2. TDX debug bit disabled (td_attributes bit0 == 0).
3. Anti-tamper binding: sha256(nonce + e2e_pubkey) must match quote report_data.
4. GPU token checks (if present):
   • x-nvidia-overall-att-result == true
   • eat_nonce == expected_report_data

Fail-close behavior: if any required check fails, proxy mode returns verification failure and does not emit a success receipt.

Receipt vs binding signature

• Binding signature: cryptographic link to upstream attestation hash + request context.
• Verification receipt: proxy-signed statement that verification policy was executed and passed.

In proxy mode, clients can verify only proxy attestation + receipt; in passthrough mode, clients can still verify full chain independently.

Implementation status

Implemented on feat/add_chutes with:

• verify_mode support (proxy default, passthrough optional)
• proxy-mode internal Chutes verification
• passthrough compatibility path
• tests under tests/app covering proxy success/failure, passthrough response, and invalid mode handling

Will follow up with any schema refinements if maintainers want stricter receipt fields/TTL semantics.