Implement publishable P2P (per-pod L4 NLB + ExternalAddress)

[bdchatham]

Problem

Sei pods today are reachable for HTTP (RPC/EVM/REST) via the existing Istio L7 path → shared NLB → private pod, but they are not publicly dialable as P2P peers over TCP/26656. This blocks validators (in a deterministic set without sentries) and advertised full nodes from being addressable by ecosystem peers, and prevents internal SNDs from peering each other across clusters.

The design (platform/docs/designs/sei-publishable-p2p-nlb.md, merged in sei-protocol/platform#714) closes this gap with an opt-in networking.tcp: {} sub-struct: when set, the SeiNode reconciler stamps a per-pod Service type=LoadBalancer (AWS NLB, target-type: ip, scheme=internet-facing, cross-zone=true), reads Service.status.loadBalancer.ingress[0].hostname, and writes SeiNode.Status.ExternalAddress = "<hostname>:26656". The planner already reads ExternalAddress into p2p.external_address at planner.go:697; this issue makes that pipeline reachable.

Impact

Unblocks:

• Validator P2P publishability — deterministic set peers each other over stable NLB DNS endpoints; combined with unconditional_peer_ids cross-config (ops, not this issue) gives stable cross-cluster validator mesh.
• Cross-SND peer discovery — new PeerSource.seinodeDeployment variant resolves sibling SND children's Status.ExternalAddress into persistent_peers, replacing the EC2-tag-only discovery for K8s-resident publishers.
• Ecosystem-parity for advertised full nodes — partner peers can list us as a persistent peer.

Harbor stays excluded until the Cilium overlay is replaced (CGNAT pod IPs aren't VPC-routable → target-type: ip fails); controller fails closed via a VPC-CIDR routability check.

Proposed approach

Six independently-shippable PRs. Sequence dependencies: PR-1 → PR-2 → PR-3 (the chain that makes the field gateable and propagated). PR-4 lands in parallel with PR-3 (independent code path; needs PR-1's types only). PR-5 lands any time before PR-6. PR-6 waits on PR-3 + PR-4 + PR-5.

PR-1 — Types layer (contract-only, no behavior change)

• api/v1alpha1/networking_types.go: add HTTP *HTTPConfig + TCP *TCPConfig to NetworkingConfig; add HTTPEnabled() accessor (legacy networking: {} → treated as http: {} for backcompat).
• api/v1alpha1/seinode_types.go: add Networking *NetworkingConfig to SeiNodeSpec; flip ExternalAddress doc-comment from "SeiNodeDeployment controller" → "SeiNode controller".
• api/v1alpha1/common_types.go: add SeiNodeDeploymentPeerSource to PeerSource union; bump XValidation CEL sum.
• CRD regen + smoke test: existing manifests (Networking-nil and legacy networking: {}) still validate.

PR-2 — SND template propagation

• internal/controller/seinodedeployment/: propagate Spec.Template.Networking into child.Spec.Networking at child-creation time (single line in the existing template-merge path).
• Envtest: SND with networking.tcp: {} in template → child SeiNodes carry Spec.Networking.TCP != nil.

PR-3 — LB Service + ExternalAddress + Service-ready gate (the heavy PR)

• internal/controller/node/external_address.go (NEW):
  • Stamp Service type=LoadBalancer (<seinode>-p2p) when Spec.Networking.TCP != nil and pod IP is inside SEI_VPC_CIDR env (VPC-CIDR routability gate).
  • Annotations: aws-load-balancer-type=nlb, scheme=internet-facing, nlb-target-type=ip, cross-zone-load-balancing-enabled=true; externalTrafficPolicy: Local; only TCP/26656.
  • Watch Service status; on loadBalancer.ingress[0].hostname populated → write Status.ExternalAddress = "<host>:26656" (bare host:port, no nodeId@).
  • Empty-host guard: if newHost == "" { return nil } — don't clear on transient absence.
  • Service-ready gate: when tcp != nil, hold STS creation until Status.ExternalAddress is written. (Without this, the first sidecar render has empty p2p.external_address; planner doesn't detect runtime override drift at planner.go:706.)
  • Opt-out cleanup: when Spec.Networking.TCP == nil, delete the Service inline.
• internal/controller/node/controller.go: confirm existing Owns(&corev1.Service{}) at :204 covers the new Service (predicate audit); add RBAC verbs if missing (create/update/delete on services).
• Manager env: SEI_VPC_CIDR plumbed via the controller Deployment (set in platform/clusters/prod/sei-k8s-controller/manager-patch.yaml — same pattern as SEI_GATEWAY_PUBLIC_DOMAIN).
• Envtest: bootstrap with tcp: {} → STS doesn't appear until Service has a (fake) hostname; opt-out path deletes the Service.

PR-4 — seinodeDeployment peer-source variant

• internal/controller/node/peers.go: extend reconcilePeers switch with the new variant; add resolveSNDPeers that lists matching SNDs → lists child SeiNodes → emits each child's Status.ExternalAddress as host:port into ResolvedPeers. The sidecar's existing CollectAndSetPeers task at internal/planner/group.go:59 prepends nodeID@ at task-build time (:26657/status query) — same path the label variant feeds. Not a new sidecar capability.
• internal/controller/node/controller.go: Watches(&SeiNodeDeployment{}, mapperFn); naive mapper (re-reconcile all SeiNodes in namespace on any SND change) is acceptable at the ~20-publishable scale; tighten to selector-matching scope when fanout becomes a concern.
• RBAC: cluster-scoped seinodedeployments get/list/watch.
• Envtest: consumer SND with peers: [{ seinodeDeployment: ... }] → resolves to producer SND children's ExternalAddress.

PR-5 — Node SG opening (in sei-protocol/platform)

• Terraform: tcp/26656 from 0.0.0.0/0 ingress on the Sei-node SG. Lands before the first publishable SND.

PR-6 — First publishable rollout (in sei-protocol/platform)

• Enable networking.tcp: {} on one advertised full-node SND.
• Verify: external nc <nlb-host> 26656; :26657/net_info shows DNS form in listen_addr; a remote node connects with <node_id>@<host>:26656.
• Validator rollout follows the same shape, one at a time across the deterministic set, with unconditional_peer_ids cross-config landed first (runbook).

Out of scope

• Config-hash STS-template annotation for automatic runtime tcp: {} add — v2. Runtime adds require a manual pod delete after Status.ExternalAddress populates (NLB hostnames don't change once allocated, so this doesn't recur).
• EIP-attached NLBs (stable IP, not just DNS) — DNS form only.
• NLB-attached SG (defense in depth) — relies on node-SG.
• Dedicated publishable EC2NodeClass / scoped SG — opens 26656 on shared node SG.
• Orphan-cleanup walker for SND scale-down events — opt-out (tcp: {} → nil) cleanup IS in PR-3; scale-down (replica drop) orphans wait for v2.
• Harbor publishability — gated on Cilium overlay → VPC-routable CNI migration. Controller fails closed.
• Validator unconditional_peer_ids cross-config — operational runbook, not a controller change.
• CometBFT sentry architecture — explicitly not adopted; deterministic validator set publishes directly.

Relevant experts

• kubernetes-specialist — PR-2, PR-3, PR-4 (controller-runtime, CRD types, envtest)
• platform-engineer — PR-3 manager env wiring, PR-5 (Terraform), PR-6 (manifest)
• sei-network-specialist — PR-6 verification (CometBFT NodeInfo + dial semantics)

References

• Design: sei-protocol/platform, docs/designs/sei-publishable-p2p-nlb.md (merged in #714)
• Planner integration point: internal/planner/planner.go:697 (commonOverrides already reads Status.ExternalAddress)
• Existing Service watch: internal/controller/node/controller.go:204
• Drift-detection limit acknowledged: internal/planner/planner.go:706 (buildRunningPlan only checks image + sidecar reapproval; ExternalAddress changes after Running don't propagate)
• AWS LBC annotation precedent: existing Istio gateway in platform/clusters/prod/gateway/gateway.yaml

One-way doors (already approved in design)

• Status.ExternalAddress format: "<host>:26656" (bare host:port; host is raw NLB *.elb.<region>.amazonaws.com, no vanity domain)
• NetworkingConfig.HTTP / TCP sub-struct field names (presence-signals)
• Legacy networking: {} interpreted as networking: { http: {} }
• SeiNodeSpec.Networking field add (additive; propagated from SND template)
• PeerSource.seinodeDeployment variant (additive; bump union CEL)
• Keep peers field name (not renamed)

[bdchatham]

Scope cuts during PR-1 review

PR #361 (PR-1) trimmed two API additions that the merged design doc proposed. The capabilities still ship — they move into PR-3's controller logic instead of carrying dedicated CRD shape:

Dropped: SeiNodeDeploymentPeerSource variant

The use case (resolve sibling SNDs' children into host:port peer entries) is already covered by the existing LabelPeerSource plus the controller-stamped sei.io/nodedeployment=<name> label on every child SeiNode.

PR-3 (resolver): extend LabelPeerSource resolution at internal/controller/node/peers.go to prefer Status.ExternalAddress over the headless Service DNS when set. Single label-selector code path, no new API, no CEL bump.

Manifest shape becomes:

peers:
  - label:
      selector: { sei.io/nodedeployment: validators-pacific-1 }
      namespace: pacific-1

Dropped: Networking field on SeiNodeSpec

Single source of truth stays on SeiNodeDeployment.Spec.Networking.

PR-3 (reconciler): the SeiNode reconciler reads the parent SND via OwnerReference for the Networking.TCP gate. No propagation step, no drift risk between parent and child. Standalone SeiNodes (no parent SND) lose publishability in v1 — only state-size-analyzer matches that shape today, no real need.

What still ships in PR-1

• NetworkingConfig.HTTP / NetworkingConfig.TCP sub-structs (presence-signals)
• HTTPEnabled() accessor with backcompat semantic (legacy networking: {} → enabled)
• Status.ExternalAddress doc-comment flip to "SeiNode controller" + bare host:port format pinned

Knock-on for downstream PRs

• PR-2 cancelled. No template propagation needed.
• PR-3 absorbs PR-4. Same package (internal/controller/node/), same RBAC delta, no separate watch wiring. Sequence is now PR-1 → PR-3 (heavy) → PR-5 (Terraform SG) → PR-6 (rollout).

Design-doc divergence (follow-up)

The merged design doc at platform/docs/designs/sei-publishable-p2p-nlb.md still describes the SeiNodeDeploymentPeerSource variant and the SeiNodeSpec.Networking propagation. The next platform PR that touches sei-controller plumbing should update those sections to reflect the actual shape that landed.