diff --git a/BACKLOG.md b/BACKLOG.md index 8837bda..f5debd3 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -14,13 +14,14 @@ Source of truth for the ShrimPK kernel project. ## Sprint Roadmap (KS73-KS80) - [x] KS73: Entity unification — EntityFrame, UUID v5, alias store, entity supersession (PR #10) -- [ ] KS74: v0.8.0-beta — recall gap fixes (NR demotion, abstention threshold), TUI dashboard, README rewrite, installer testing -- [ ] KS75: Store-time contradiction detection -- [ ] KS76: Memory import — cold start solver, 4+ parsers (Claude Code, ChatGPT, Obsidian, Mem0) -- [ ] KS77: KU-3 fix + remaining recall fixes (90% gate) -- [ ] KS78: Public launch preparation -- [ ] KS79: Context compression — LLMLingua-2 ONNX at store time -- [ ] KS80: Retroactive link re-scoring + sleep replay +- [x] KS74: LME micro-benchmark suite + evaluation pipeline +- [x] KS75: Runtime-configurable embedding model — EmbeddingProvider trait, 10 fastembed + OpenAI API (PR #14) +- [x] KS76: Universal prompt + temporal boost + importance scoring (PR #15) +- [x] KS77: KU-3 fix + abstention calibration — 19/20 seeded, 5/5 abstention (PR #17) +- [x] KS77b: Design system foundation + P0 graph polish — Tailwind 4.0 @theme, 6 components, 10 micro-improvements (PR #18) +- [ ] KS78: Critical bug fixes — persistence format (#16), recency test panic (#13), supersession demotion (#11) +- [ ] KS79: Multi-resolution retrieval — PPR-weighted Hebbian, fallback cascade, retrieval modes +- [ ] KS80: Memory lifecycle — merge operation, soft-deletion compaction, multi-granularity storage --- @@ -52,7 +53,7 @@ Source of truth for the ShrimPK kernel project. - [ ] f16 quantization for vision/speech — SHRM v3, ~50% disk/RAM savings, f32 promotion at query time - [ ] Band-limited resampling — replace resample_linear() with rubato crate. Correctness bug: aliasing at 48→16kHz - [ ] BuiltinConsolidator — bundled extraction model, zero Ollama dependency for consolidation quality -- [ ] Configurable embedding provider — EmbeddingProvider trait, 10 fastembed models + OpenAI API (KS75 — DONE) +- [x] Configurable embedding provider — EmbeddingProvider trait, 10 fastembed models + OpenAI API (KS75, PR #14) ## MEDIUM — Graph & Entity @@ -144,4 +145,4 @@ Source of truth for the ShrimPK kernel project. --- -*Last updated: 2026-04-09* +*Last updated: 2026-04-10* diff --git a/CODEX.md b/CODEX.md new file mode 100644 index 0000000..716391f --- /dev/null +++ b/CODEX.md @@ -0,0 +1,75 @@ +# ShrimPK Kernel — Codex Instructions + +You are a **code reviewer and secondary QA** for the ShrimPK kernel project. +Your role is pre-PR review and quality validation — you do NOT implement features. + +## Project Overview + +Push-based AI memory kernel in Rust. Apache 2.0. v0.7.0. +11 crates + CLI in a Cargo workspace. + +## Your Responsibilities + +### Pre-PR Code Review +- Review all code changes before PR submission +- Flag logic errors, off-by-one bugs, panic paths, unwrap misuse +- Check for OWASP top 10 vulnerabilities (injection, XSS in viz, etc.) +- Verify test coverage: every new public function needs a test +- Ensure `cargo clippy --workspace` passes clean +- Ensure `cargo fmt --check` passes + +### Secondary QA +- Run `cargo test --workspace` and verify all tests pass +- Run the seeded micro-benchmark: `cargo test -p shrimpk-memory -- benchmark --nocapture` +- Verify daemon health: `curl http://localhost:11435/health` +- Check for regressions in benchmark scores (current: 19/20 seeded, 5/5 abstention, 3/3 NR) + +## Workspace Structure + +| Crate | Purpose | +|-------|---------| +| `shrimpk-core` | Types: MemoryEntry, EchoResult, EchoConfig, Modality | +| `shrimpk-memory` | Engine: EchoEngine, embedding, LSH, Bloom, Hebbian, labels, FSRS decay, ACT-R | +| `shrimpk-daemon` | HTTP server: axum, proxy, routes (/health, /debug, /v1/chat/completions) | +| `shrimpk-mcp` | MCP server (stdio): store, echo, forget, stats, graph navigation | +| `shrimpk-context` | ContextAssembler: token-budgeted prompt compilation | +| `shrimpk-router` | CascadeRouter: provider routing (NOT wired in daemon yet) | +| `shrimpk-security` | PII masking (stub) | +| `shrimpk-kernel` | Facade crate re-exporting core + memory + context | +| `shrimpk-python` | PyO3 bindings (maturin) | +| `shrimpk-ros2` | ROS2 bridge (stub) | +| `shrimpk-tray` | Windows system tray (win32) | +| `cli/` | CLI binary: store, echo, status, explore (ratatui TUI) | + +## Review Checklist + +1. **Correctness**: Does the code do what it claims? Test edge cases mentally. +2. **Safety**: No `unwrap()` on user input. `expect()` only with clear message. No panics in daemon/MCP paths. +3. **Performance**: No unnecessary allocations in hot paths (echo, store). Watch for `clone()` abuse. +4. **Concurrency**: Mutex/RwLock usage correct? No deadlock patterns? `block_in_place` for sync-in-async? +5. **Tests**: New code has tests. Tests are deterministic (seeded RNG where needed). +6. **Clippy**: Zero warnings. `#[allow(clippy::*)]` needs justification comment. +7. **Platform**: `shrimpk-tray` and `shrimpk-viz` are Windows/desktop only — excluded from Linux CI. + +## What NOT to Do + +- Do not implement features or write code +- Do not merge PRs +- Do not modify CLAUDE.md or CODEX.md +- Do not run benchmarks that require Ollama (heavy compute) + +## Build & Test Commands + +```bash +cargo build --workspace # debug build +cargo test --workspace # all tests +cargo clippy --workspace -- -D warnings # lint +cargo fmt --check # format check +``` + +## Current Benchmark Baselines + +- Seeded micro-benchmark: 19/20 (95%) +- Abstention: 5/5 (100%) +- Negative recall: 3/3 (100%) +- LME-S 500q (GPT-4o judge): 24.2% baseline, KS76/77 eval pending diff --git a/crates/shrimpk-memory/src/persistence.rs b/crates/shrimpk-memory/src/persistence.rs index 273b343..5157604 100644 --- a/crates/shrimpk-memory/src/persistence.rs +++ b/crates/shrimpk-memory/src/persistence.rs @@ -490,7 +490,8 @@ pub fn load_binary(path: &Path) -> Result { Ok(store) } else { Err(ShrimPKError::Persistence(format!( - "Unsupported format version: {}", + "Unsupported format version: {} (supported: 1, 2). \ + If your data was written by a newer ShrimPK version, update your binary.", version_byte ))) } @@ -1194,11 +1195,18 @@ mod tests { let result = load_binary(&path); assert!(result.is_err(), "Should reject wrong version"); + let err_msg = result.unwrap_err().to_string(); assert!( - result - .unwrap_err() - .to_string() - .contains("Unsupported format version") + err_msg.contains("Unsupported format version: 99"), + "Should include the version number: {err_msg}" + ); + assert!( + err_msg.contains("supported: 1, 2"), + "Should list supported versions: {err_msg}" + ); + assert!( + err_msg.contains("update your binary"), + "Should hint at updating the binary: {err_msg}" ); }