Entity unification: subjects, triples, supersession, labels, Hebbian all doing entity work independently

[Liorrr]

Problem

ShrimPK has 9 separate mechanisms that independently approximate entity identification without a unifying layer:

1. subject field on MemoryEntry (core/memory.rs:234) — extracted per child fact
2. Entity labels (entity:* prefix in labels.rs) — keyword-based, not linked to entities
3. Triples (subject, predicate, object) — stored per memory, disconnected from other entity refs
4. fix_degenerate_subject() (consolidation.rs:700) — 100-line heuristic to repair "the user"/"I"/"me"
5. child_topic_matches_query() (echo.rs:3008) — label/subject gate, fallback-based
6. Subject diversity cap (echo.rs:3283) — per-(subject, topic) result limiting
7. subjects_overlap() (consolidation.rs:1225) — case-insensitive subject comparison for supersession
8. detect_relationship() (consolidation.rs:897) — regex relationship type extraction
9. Hebbian co-activation (hebbian.rs) — typed edges (WorksAt, LivesIn, etc.) but not entity-anchored

These mechanisms don't talk to each other. "Sam", "Sam Torres", "the user", and "I" are treated as different subjects across different mechanisms. No centralized entity registry exists.

Impact

• Supersession fails when subject heuristics disagree (e.g., "the user" vs "Sam" in different facts)
• No entity profiles — can't answer "tell me everything about Sam" without full embedding scan
• Contradiction detection impossible — can't check for conflicting facts about the same entity without knowing they're the same entity
• GDPR deletion incomplete — can't find all traces of an entity across subjects, triples, labels, and Hebbian edges
• 6 roadmap features blocked: contradiction detection, tombstone propagation, faithfulness scoring, collaborative memory, memory-type routing, entity-centric retrieval

Proposed Solution: EntityFrame

A lightweight entity registry (EntityFrame) that absorbs existing mechanisms rather than adding another layer:

• Replaces: fix_degenerate_subject() (~100 lines), subjects_overlap() (~30 lines), store entity_index
• Absorbs: subject field becomes entity_id: Option<EntityId>, entity: labels generated from frame, triple subjects resolve to EntityId
• Untouched: Hebbian graph structure, non-entity labels, scoring pipeline, confidence/quality gates

Net complexity: ~+250 new lines, ~-300 removed heuristics. The system gets simpler.

Store-time detection: Aho-Corasick over known aliases (zero LLM calls). Entity-less memories (pure events, moods) get no entity assignment — not force-fitted.

Design Brief

Full design brief with EntityFrame structure, open questions, and roadmap impact at:
Obsidian Vault/ShrimPK Kernel/Design Brief — Entity Identification.md

Related Research

• Zep/Graphiti: NER + LLM entity extraction + temporal validity per entity edge
• Mem0: User profile building via fact extraction
• MIRIX (arXiv 2507.07957): "core biography" memory type per entity
• A-MEM (NeurIPS 2025): Zettelkasten notes with retroactive link updates on entity changes

Labels

• enhancement
• architecture
• consolidation

[Liorrr]

Team Brainstorm Results (6 agents, KS72)

Phase 1 (low risk, high value — next sprint)

Make supersession use MemoryEntry.subject instead of subjects_overlap() regex. We already compute the subject via LLM + fix_degenerate_subject() — then ignore it for the one operation where it matters most. Falls back to regex if subject is None.

Phase 2 (medium risk)

Decouple entity values from RelationshipType. Move WorksAt(String) to WorksAt + separate entity field on Hebbian edge. Avoids duplicating entity strings across edge and index. Needs serde migration for existing .hebbian files.

Phase 3 (big refactor — multi-sprint)

Full EntityRegistry with resolution, aliases, dedup. Touches persistence, scoring, graph, and MCP. Design brief at Obsidian/ShrimPK Kernel/Design Brief — Entity Identification.md.

Key redundancies identified

• MemoryEntry.subject vs Triple.subject — same data, different extraction paths, can diverge
• RelationshipType::WorksAt("Google") vs entity_index["google"] — same entity stored twice with different normalization
• subjects_overlap() ignores MemoryEntry.subject and re-extracts via regex
• detect_relationship() runs twice on the same fact (triples + supersession)

Entity resolution cascade (from research)

1. Rule-based: capitalized nouns, pronouns→USER, tool-name lookup (~0ms)
2. Embedding cosine ≥ 0.78 + token overlap → merge
3. LLM only for 0.5-0.78 borderline (async)
4. gline-rs (Rust ONNX GLiNER, 188MB) available for zero-shot NER

Benchmark gaps to address

• Negative tests: superseded facts NOT in top-3
• Decay tests: old memories score lower
• Relational tests: multi-hop ("What does Sam's partner do?")
• Collision tests: Sam vs Jordan attribute separation

Full brainstorm notes: Obsidian/ShrimPK Kernel/KS72 Team Brainstorm — Entity & Architecture.md