Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,14 @@ Source of truth for the ShrimPK kernel project.
## Sprint Roadmap (KS73-KS80)

- [x] KS73: Entity unification — EntityFrame, UUID v5, alias store, entity supersession (PR #10)
- [ ] KS74: v0.8.0-beta — recall gap fixes (NR demotion, abstention threshold), TUI dashboard, README rewrite, installer testing
- [ ] KS75: Store-time contradiction detection
- [ ] KS76: Memory import — cold start solver, 4+ parsers (Claude Code, ChatGPT, Obsidian, Mem0)
- [ ] KS77: KU-3 fix + remaining recall fixes (90% gate)
- [ ] KS78: Public launch preparation
- [ ] KS79: Context compression — LLMLingua-2 ONNX at store time
- [ ] KS80: Retroactive link re-scoring + sleep replay
- [x] KS74: LME micro-benchmark suite + evaluation pipeline
- [x] KS75: Runtime-configurable embedding model — EmbeddingProvider trait, 10 fastembed + OpenAI API (PR #14)
- [x] KS76: Universal prompt + temporal boost + importance scoring (PR #15)
- [x] KS77: KU-3 fix + abstention calibration — 19/20 seeded, 5/5 abstention (PR #17)
- [x] KS77b: Design system foundation + P0 graph polish — Tailwind 4.0 @theme, 6 components, 10 micro-improvements (PR #18)
- [ ] KS78: Critical bug fixes — persistence format (#16), recency test panic (#13), supersession demotion (#11)
- [ ] KS79: Multi-resolution retrieval — PPR-weighted Hebbian, fallback cascade, retrieval modes
- [ ] KS80: Memory lifecycle — merge operation, soft-deletion compaction, multi-granularity storage

---

Expand Down Expand Up @@ -52,7 +53,7 @@ Source of truth for the ShrimPK kernel project.
- [ ] f16 quantization for vision/speech — SHRM v3, ~50% disk/RAM savings, f32 promotion at query time
- [ ] Band-limited resampling — replace resample_linear() with rubato crate. Correctness bug: aliasing at 48→16kHz
- [ ] BuiltinConsolidator — bundled extraction model, zero Ollama dependency for consolidation quality
- [ ] Configurable embedding provider — EmbeddingProvider trait, 10 fastembed models + OpenAI API (KS75 — DONE)
- [x] Configurable embedding provider — EmbeddingProvider trait, 10 fastembed models + OpenAI API (KS75, PR #14)

## MEDIUM — Graph & Entity

Expand Down Expand Up @@ -144,4 +145,4 @@ Source of truth for the ShrimPK kernel project.

---

*Last updated: 2026-04-09*
*Last updated: 2026-04-10*
75 changes: 75 additions & 0 deletions CODEX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# ShrimPK Kernel — Codex Instructions

You are a **code reviewer and secondary QA** for the ShrimPK kernel project.
Your role is pre-PR review and quality validation — you do NOT implement features.

## Project Overview

Push-based AI memory kernel in Rust. Apache 2.0. v0.7.0.
11 crates + CLI in a Cargo workspace.

## Your Responsibilities

### Pre-PR Code Review
- Review all code changes before PR submission
- Flag logic errors, off-by-one bugs, panic paths, unwrap misuse
- Check for OWASP top 10 vulnerabilities (injection, XSS in viz, etc.)
- Verify test coverage: every new public function needs a test
- Ensure `cargo clippy --workspace` passes clean
- Ensure `cargo fmt --check` passes

### Secondary QA
- Run `cargo test --workspace` and verify all tests pass
- Run the seeded micro-benchmark: `cargo test -p shrimpk-memory -- benchmark --nocapture`
- Verify daemon health: `curl http://localhost:11435/health`
- Check for regressions in benchmark scores (current: 19/20 seeded, 5/5 abstention, 3/3 NR)

## Workspace Structure

| Crate | Purpose |
|-------|---------|
| `shrimpk-core` | Types: MemoryEntry, EchoResult, EchoConfig, Modality |
| `shrimpk-memory` | Engine: EchoEngine, embedding, LSH, Bloom, Hebbian, labels, FSRS decay, ACT-R |
| `shrimpk-daemon` | HTTP server: axum, proxy, routes (/health, /debug, /v1/chat/completions) |
| `shrimpk-mcp` | MCP server (stdio): store, echo, forget, stats, graph navigation |
| `shrimpk-context` | ContextAssembler: token-budgeted prompt compilation |
| `shrimpk-router` | CascadeRouter: provider routing (NOT wired in daemon yet) |
| `shrimpk-security` | PII masking (stub) |
| `shrimpk-kernel` | Facade crate re-exporting core + memory + context |
| `shrimpk-python` | PyO3 bindings (maturin) |
| `shrimpk-ros2` | ROS2 bridge (stub) |
| `shrimpk-tray` | Windows system tray (win32) |
| `cli/` | CLI binary: store, echo, status, explore (ratatui TUI) |

## Review Checklist

1. **Correctness**: Does the code do what it claims? Test edge cases mentally.
2. **Safety**: No `unwrap()` on user input. `expect()` only with clear message. No panics in daemon/MCP paths.
3. **Performance**: No unnecessary allocations in hot paths (echo, store). Watch for `clone()` abuse.
4. **Concurrency**: Mutex/RwLock usage correct? No deadlock patterns? `block_in_place` for sync-in-async?
5. **Tests**: New code has tests. Tests are deterministic (seeded RNG where needed).
6. **Clippy**: Zero warnings. `#[allow(clippy::*)]` needs justification comment.
7. **Platform**: `shrimpk-tray` and `shrimpk-viz` are Windows/desktop only — excluded from Linux CI.

## What NOT to Do

- Do not implement features or write code
- Do not merge PRs
- Do not modify CLAUDE.md or CODEX.md
- Do not run benchmarks that require Ollama (heavy compute)

## Build & Test Commands

```bash
cargo build --workspace # debug build
cargo test --workspace # all tests
cargo clippy --workspace -- -D warnings # lint
cargo fmt --check # format check
```

## Current Benchmark Baselines

- Seeded micro-benchmark: 19/20 (95%)
- Abstention: 5/5 (100%)
- Negative recall: 3/3 (100%)
- LME-S 500q (GPT-4o judge): 24.2% baseline, KS76/77 eval pending
18 changes: 13 additions & 5 deletions crates/shrimpk-memory/src/persistence.rs
Original file line number Diff line number Diff line change
Expand Up @@ -490,7 +490,8 @@ pub fn load_binary(path: &Path) -> Result<EchoStore> {
Ok(store)
} else {
Err(ShrimPKError::Persistence(format!(
"Unsupported format version: {}",
"Unsupported format version: {} (supported: 1, 2). \
If your data was written by a newer ShrimPK version, update your binary.",
version_byte
)))
}
Expand Down Expand Up @@ -1194,11 +1195,18 @@ mod tests {

let result = load_binary(&path);
assert!(result.is_err(), "Should reject wrong version");
let err_msg = result.unwrap_err().to_string();
assert!(
result
.unwrap_err()
.to_string()
.contains("Unsupported format version")
err_msg.contains("Unsupported format version: 99"),
"Should include the version number: {err_msg}"
);
assert!(
err_msg.contains("supported: 1, 2"),
"Should list supported versions: {err_msg}"
);
assert!(
err_msg.contains("update your binary"),
"Should hint at updating the binary: {err_msg}"
);
}

Expand Down
Loading