Skip to content

Commit 21b61eb

Browse files
committed
docs(pr-x12): address PR #197 CodeRabbit review — 7 open items
Arithmetic errors - R-11 (canon-resolutions): 132,710 was 8×8 leaves/frame, not 64×64 CTUs (which is 2,040). Relabelled the per-CTU breakdown table and budget derivation to per-leaf; clarified the 1 CTU = ~64 leaves at max split depth relationship. Latency narrative unchanged. - R-12 (canon-resolutions): per-CTU flush rate at 4K was 80,000/sec but 2,040 CTUs × 60 fps = 122,400/sec; 1080p figure refined to 30,600/sec (510 CTUs × 60). - x266 §6 bandwidth: HEVC at 4K/60 from 1080p/30 scales as 4 (res) × 2 (fps) × 6.25 MB = 50 MB, not 4 × 4 × 6.25 = 100 MB. PR-X12 wins by ~6× (not 12×); crossover ~1.3× (not 3×). Bits 14-15 consistency (3 sites reconciled to R-2 canon) - merged-canon M:E-J §3 formalisation note: replaced stale 'bits 14-15 = leaf_size' with R-2's bit 15 = UNIVERSAL inter-tier / bit 14 = CONSUMER-TYPED via ConsumerProfile. Leaf size lives in Ctu<const N> at the type level (M:E-G), not in header bits. - x266 §5 header layout: same R-2 alignment for the HEVC-compatible header block. - anti-neural §6: enhancement-layer flag moved from per-leaf bit 14 (claimed by R-2 consumer-typed demux) to the frame header reserved area (alongside ConsumerProfile and FlushUnit per R-2/R-12). Path / citation corrections (cam-pq-sigker-dn-tree doc) - Absolute /home/user/ndarray/... paths replaced with repo-relative src/hpc/... since this repo IS ndarray. - /home/user/lance-graph/... paths marked as external repo (adaworldapi/lance-graph), not resolvable in this checkout. - Hambly-Lyons 2010 citation now includes arXiv:math/0507536 and Annals of Mathematics 171(1):109-167. All claims re-verified by reading each affected file in full (canon-resolutions 1281 lines, x266 321, merged-canon 647, anti-neural 337, cam-pq 395) before edits — the same protocol applied to the bgz-jc proofread. https://claude.ai/code/session_01HbqooFZHAjaUtFEzhA1R2u
1 parent f1d68e6 commit 21b61eb

5 files changed

Lines changed: 58 additions & 32 deletions

.claude/knowledge/pr-x12-anti-neural-lookup-inversion.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ For these use cases, the right architecture is a **layered codec**:
234234
1. **Base layer:** PR-X12 frozen-lookup codec for the bits-actually-transmitted
235235
2. **Enhancement layer:** NN generative refinement at the decoder (optional, off by default)
236236

237-
The base layer guarantees fidelity bounded by Shannon. The enhancement layer provides perceptual hallucination when the user opts in. PR-X12's wire format reserves a single bit (M:E-J bit 14 currently used for leaf_size; one of the reserved bits in future revisions) for the "enhancement layer available" flag.
237+
The base layer guarantees fidelity bounded by Shannon. The enhancement layer provides perceptual hallucination when the user opts in. PR-X12's wire format reserves a single bit in the **frame header** (alongside `ConsumerProfile` and `FlushUnit` per R-2 / R-12) for the "enhancement layer available" flag — not in the per-leaf 16-bit header, whose bit 14 is already claimed by R-2's consumer-typed demux and whose bit 15 is the universal inter-tier reference.
238238

239239
This is also the right architecture for high-stakes content (legal, medical, scientific): always run the base layer, never run the enhancement layer. Determinism preserved.
240240

.claude/knowledge/pr-x12-cam-pq-sigker-dn-tree-substrate-bindings.md

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
### 1.1 What it is
1919

20-
**Location:** `/home/user/ndarray/src/hpc/cam_pq.rs`
20+
**Location:** `src/hpc/cam_pq.rs` (this repo)
2121

2222
**Algorithm:** Content-Addressable Memory (CAM) + Product Quantization (PQ). Unifies FAISS PQ6×8 (48-bit fingerprints, 6 subspaces × 256 centroids each) with CLAM 48-bit archetypes into a single codec.
2323

@@ -108,7 +108,7 @@ The codebook implementation is `cam_pq::CamCodebook`. The four policy variants c
108108

109109
### 2.1 What it is
110110

111-
**Location:** `/home/user/lance-graph/crates/sigker/`
111+
**Location:** `crates/sigker/` in the external `adaworldapi/lance-graph` repo (not in this `ndarray` repo)
112112

113113
**Algorithm:** Path-signature representations for sequential / path-structured data. Implements Chen-Lyons signatures S(X) = (1, ∫dX, ∫∫dX⊗dX, …) up to depth N, with shuffle-product algebra and proven uniqueness.
114114

@@ -134,7 +134,7 @@ pub struct CodecRouteSigker { /* lance-graph codec routing integration */ }
134134
|---|---|---|
135135
| Chen, "Iterated integrals and exponential homomorphisms" | 1957 | Original signature construction |
136136
| Lyons, "Differential equations driven by rough signals" | 1998 | Rough path theory, signature universal approximator |
137-
| Hambly-Lyons, "Uniqueness for the signature of a path of bounded variation" | 2010 | **Theorem 4: signatures uniquely determine paths up to tree-like equivalence** |
137+
| Hambly-Lyons, "Uniqueness for the signature of a path of bounded variation" (**arXiv:math/0507536**, Annals of Mathematics 171(1):109–167) | 2010 | **Theorem 4: signatures uniquely determine paths up to tree-like equivalence** |
138138
| Salvi-Cass-Foster-Lyons-Lemercier | 2020 | **arXiv:2006.14794** — Goursat-PDE solver for signature kernel, O(T₁·T₂·d), no signature materialization |
139139
| Cuchiero-Schmocker-Teichmann | 2021 | **Randomized signature universality**: any continuous path-functional ≈ linear combo of randomized-signature coordinates |
140140

@@ -206,7 +206,7 @@ This unlocks: **path-structured codec lanes** in Plan G (audio waveforms, time-s
206206

207207
### 3.1 dn_tree — quaternary plastic memory
208208

209-
**Location:** `/home/user/ndarray/src/hpc/dn_tree.rs`
209+
**Location:** `src/hpc/dn_tree.rs` (this repo)
210210

211211
**Algorithm:** Quaternary hierarchical bitmap summary tree for plastic graph traversal. Adapted from "On Demand Memory Specialization for Distributed Graph Processing" (2013). Properties:
212212

@@ -222,7 +222,7 @@ This unlocks: **path-structured codec lanes** in Plan G (audio waveforms, time-s
222222

223223
### 3.2 merkle_tree — integrity proof for CogRecord regions
224224

225-
**Location:** `/home/user/ndarray/src/hpc/merkle_tree.rs`
225+
**Location:** `src/hpc/merkle_tree.rs` (this repo)
226226

227227
**Algorithm:** 8-Kbit Merkle tree built from CogRecord regions as a compressed searchable proxy. Properties:
228228

@@ -376,15 +376,16 @@ This doc (#4) and the bgz/jc doc (#3) are the ones that ground PR-X12 in working
376376
- **GGUF lens (activation-aware RDO claim):** `pr-x12-gguf-llm-weights-encoding.md` §5 — supported by G-1 closure
377377
- **Anti-neural lens (lookup-table cost analysis):** `pr-x12-anti-neural-lookup-inversion.md` §3 — supported by G-4 + G-5 closure
378378
- **Multi-arch lens (determinism + integrity):** `pr-x12-woa-multiarch-orchestration.md` §6 — supported by G-4 + G-7 closure
379-
- **Source code references:**
380-
- `/home/user/ndarray/src/hpc/cam_pq.rs` — the codebook trainer
381-
- `/home/user/ndarray/src/hpc/dn_tree.rs` — quaternary plastic memory
382-
- `/home/user/ndarray/src/hpc/merkle_tree.rs` — Blake3-48-bit Merkle
383-
- `/home/user/lance-graph/crates/sigker/` — Chen-Lyons signatures
384-
- `/home/user/lance-graph/crates/sigker/src/``signature_kernel_pde`, `RandomizedSignature`, `CodecRouteSigker`
385-
- `/home/user/lance-graph/crates/jc/src/hambly_lyons.rs` — Pillar 11 (active under `--features hambly-lyons`; DEFERRED only in default zero-dep build)
386-
- `/home/user/lance-graph/crates/jc/src/pflug.rs` — Pillar 10 (nested-distance Lipschitz on Sigma DN-trees, certifies CAM-PQ)
387-
- `/home/user/lance-graph/crates/bgz-tensor/src/adaptive_codec.rs` — cam_pq imports
379+
- **Source code references (in this repo `adaworldapi/ndarray`):**
380+
- `src/hpc/cam_pq.rs` — the codebook trainer
381+
- `src/hpc/dn_tree.rs` — quaternary plastic memory
382+
- `src/hpc/merkle_tree.rs` — Blake3-48-bit Merkle
383+
- **Source code references (external repo `adaworldapi/lance-graph`):**
384+
- `crates/sigker/` — Chen-Lyons signatures
385+
- `crates/sigker/src/``signature_kernel_pde`, `RandomizedSignature`, `CodecRouteSigker`
386+
- `crates/jc/src/hambly_lyons.rs` — Pillar 11 (active under `--features hambly-lyons`; DEFERRED only in default zero-dep build)
387+
- `crates/jc/src/pflug.rs` — Pillar 10 (nested-distance Lipschitz on Sigma DN-trees, certifies CAM-PQ)
388+
- `crates/bgz-tensor/src/adaptive_codec.rs` — cam_pq imports
388389
- **arXiv anchors for sigker:**
389390
- **2006.14794** (Salvi-Cass-Foster-Lyons-Lemercier 2020) — Goursat PDE for signature kernel
390391
- Hambly-Lyons 2010 — signature uniqueness theorem

.claude/knowledge/pr-x12-substrate-canon-resolutions.md

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -767,11 +767,17 @@ to the budget.
767767
```text
768768
4K = 3840 × 2160 = 8.3 M pixels
769769
60 fps = 16.67 ms/frame
770-
At 64×64 CTU: 132,710 CTUs/frame
771-
Per-CTU budget: 16.67 / 132710 = 125 ns/CTU
770+
At 8×8 leaf granularity (HEVC's smallest CU; the unit at which the
771+
encoder's inner-loop work is paid):
772+
132,710 leaves/frame
773+
(= 2,040 CTUs/frame at 64×64, × ~64
774+
leaves/CTU at maximum split depth;
775+
130,560 from clean 3840·2160/64, with
776+
~1.6 % bias for chroma alignment)
777+
Per-leaf budget: 16.67 ms / 132,710 = 125 ns/leaf
772778
```
773779

774-
**Encoder per-CTU breakdown (scalar reference, current):**
780+
**Encoder per-leaf breakdown (scalar reference, current):**
775781

776782
| Stage | Scalar cost | SIMD-batched target |
777783
|-------|-------------|---------------------|
@@ -781,12 +787,12 @@ Per-CTU budget: 16.67 / 132710 = 125 ns/CTU
781787
| transform (A4, 8×8 DCT-II butterfly) | ~30 ns | ~30 ns |
782788
| quantize (i8 round) | ~5 ns | ~5 ns |
783789
| rANS encode (A7) | ~40 ns | ~40 ns |
784-
| **Total per-CTU** | **~960 ns** | **~210 ns** |
790+
| **Total per-leaf** | **~960 ns** | **~210 ns** |
785791

786-
**At scalar reference (960 ns/CTU): 4K @ 60 fps requires 132710 ×
792+
**At scalar reference (960 ns/leaf): 4K @ 60 fps requires 132,710 ×
787793
960 ns = 127 ms/frame. Misses 60 fps by 7.6×.**
788794

789-
**At SIMD-batched (210 ns/CTU): 132710 × 210 ns = 28 ms/frame. Misses
795+
**At SIMD-batched (210 ns/leaf): 132,710 × 210 ns = 28 ms/frame. Misses
790796
60 fps by 1.7×; needs further work but in the same order of magnitude.**
791797

792798
**To hit 60 fps 4K real-time** requires the SIMD-batched-encode path
@@ -797,8 +803,8 @@ reference only.
797803
**Implication for Plan G.** The `--mode video` threshold (R-4)
798804
includes a latency assertion: total encode time for the Big Buck Bunny
799805
1080p clip must complete within (clip duration × 0.5). At 1080p that's
800-
33,825 CTUs/frame × 210 ns × 30 fps = ~213 ms/sec, well within budget.
801-
4K is the stretch target.
806+
~32,400 leaves/frame × 210 ns × 30 fps = ~204 ms/sec, well within
807+
budget. 4K is the stretch target.
802808

803809
**Cite as R-11 in any encoder-path PR description; the latency
804810
budget is the gate that determines whether SIMD-batched encode is P0
@@ -813,13 +819,14 @@ Different answers make Plan A8 substantially different shapes.
813819

814820
**Resolution.** Commit per-CTU as the default; per-bucket for Plan F.
815821

816-
**Per-CTU flush (committed default):**
822+
**Per-CTU flush (committed default; CTU = 64×64 cells, so 4096 cells/CTU,
823+
2,040 CTUs/frame at 4K and ~510 CTUs/frame at 1080p):**
817824

818825
```text
819826
Buffer size: ~12 KB per CTU
820827
= 4096 cells × avg 3 bytes (mode-distribution per R-10)
821-
Flush rate: ~80,000 flushes/sec at 4K 60 fps (132710 CTU/frame × 60)
822-
~30,000 flushes/sec at 1080p 60 fps
828+
Flush rate: ~122,400 flushes/sec at 4K 60 fps (2,040 CTUs/frame × 60)
829+
~30,600 flushes/sec at 1080p 60 fps (510 CTUs/frame × 60)
823830
Latency: sub-ms per CTU; consumer can start decoding the first
824831
CTU before encoder finishes the frame
825832
```

.claude/knowledge/pr-x12-substrate-merged-canon.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -278,7 +278,18 @@ pub trait PredictiveSignal {
278278

279279
### M:E-J — The reserved header bits 14-15 carry causal-edge metadata for free
280280

281-
> [Formalised post-merge as **R-2**: 16-bit header bit layout pinned — bits 0-1 = `header_kind`, bits 2-13 = `basin_index`, bits 14-15 = `leaf_size ∈ {8,16,32,64}` (which subsumes M:E-G's `Ctu<const N>`). The causal-tier reading below remains valid but is now the *interpretation*, not the on-wire layout — see R-2 and R-8.]
281+
> [Formalised post-merge as **R-2**: 16-bit header bit layout pinned —
282+
> bits 0-1 = `header_kind`, bits 2-13 = `basin_index`,
283+
> **bit 15 = UNIVERSAL "has inter-tier reference"** (identical across
284+
> all four consumers; A3-inter cross-tier link),
285+
> **bit 14 = CONSUMER-TYPED via the frame header's `ConsumerProfile`
286+
> tag** (cognitive: Pearl-rung high bit; video: reserved=0;
287+
> splat: LOD-cascade-source flag; gradient: worker-shard parity).
288+
> Leaf size (8/16/32/64) is encoded structurally via M:E-G's
289+
> `Ctu<const N>` at the type level, NOT in header bits 14-15. The
290+
> causal-tier reading below is the historical motivation for bit 14;
291+
> R-2 generalises it to the four-consumer demux. See
292+
> `pr-x12-substrate-canon-resolutions.md` §R-2.]
282293
283294
A's E-15 (reserved bits 14-15 are inter-tier link) + A's T-22 (causal-edge v2 mantissa: Intervention=+6, Counterfactual=-6):
284295

.claude/knowledge/pr-x12-x266-3dgs-spacetime-upscaling.md

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,13 @@ Building on M:E-J's 16-bit header layout (header_kind ∈ {Skip, Merge, Delta, E
163163
HEVC-compatible PR-X12 header (16 bits, R-2):
164164
bits 0-1: header_kind {Skip, Merge, Delta, Escape}
165165
bits 2-13: basin_index (12 bits, M:E-J)
166-
bits 14-15: leaf_size ∈ {8, 16, 32, 64}
166+
bit 14: CONSUMER-TYPED (semantic per frame-header `ConsumerProfile`;
167+
cognitive: Pearl-rung high bit; video: reserved=0;
168+
splat: LOD-cascade-source flag; gradient: worker-shard parity)
169+
bit 15: UNIVERSAL "has inter-tier reference" (A3-inter); identical
170+
across all four consumers
171+
NOTE: leaf-size (8/16/32/64) is encoded structurally via `Ctu<const N>`
172+
(M:E-G) at the type level, not via header bits.
167173
168174
x266 extension (NOT in PR-X12 scope, future):
169175
bits 0-1: header_kind, now 4 variants
@@ -214,16 +220,17 @@ PR-X12 + 3DGS anchor (single anchor for the clip):
214220
→ HEVC wins by ~25% for native (1080p, 30 fps) playback.
215221
216222
BUT for 4K @ 60 fps playback:
217-
HEVC: re-encode at 4K/60fps target = 4 × 4 × 6.25 = 100 MB
218-
(or super-res upscaling at decode = 6.25 MB + neural inference)
223+
HEVC: re-encode at 4K/60fps target = 4 (res) × 2 (fps) × 6.25 = 50 MB
224+
(4× pixel scaling × 2× framerate scaling × 6.25 MB native bitrate;
225+
or super-res upscaling at decode = 6.25 MB + neural inference)
219226
PR-X12 + 3DGS: same 8.3 MB
220227
decoder rasterizes at (4K, 60 fps); the math is in the scene
221228
222-
→ PR-X12 wins by 12× for high-resolution playback,
229+
→ PR-X12 wins by ~6× for high-resolution playback,
223230
AND playback is deterministic (no neural model versioning).
224231
```
225232

226-
**Where the crossover sits:** PR-X12 + 3DGS becomes a win when the playback target (W × H × fps) exceeds the encode target by ~3×. At 1× (native), HEVC is a hair cheaper. At 12× (4K@60 from 1080p@24), PR-X12 dominates.
233+
**Where the crossover sits:** PR-X12 + 3DGS becomes a win when the playback target (W × H × fps) exceeds the encode target by ~1.3× (the point at which HEVC's re-encoded size crosses the fixed 8.3 MB PR-X12 budget). At 1× (native), HEVC is a hair cheaper. At 8× pixel-bandwidth (4K@60 from 1080p@30), PR-X12 dominates by ~.
227234

228235
This matches the intuition that **3DGS is a scene model**, not a frame model — its compression ratio improves with resolution, while HEVC's degrades.
229236

0 commit comments

Comments
 (0)