Support filtering

[mindsocket]

We have a schema, and it is open for extension.

It should be possible to filter an OST to show more or less detail.

Less depth for a simplified view of just the top layers
Less breadth to focus in on an area
Less density to exclude ideas that aren't in focus (eg leave out status: identified

I'm hoping we can define a simple filter language or find an existing one that's human friendly and works with json. Todoist filters are a good reference - they support basic boolean logic(| and & and parens), some keywords that translate to specific filters (eg "today", "@Label"). Another source of inspiration might be something akin to CSS or XPath selectors.

Let's do a top down and bottom up investigation of options - top down for good filter approaches, bottom up for existing low level json filtering - then see where they meet.

We'll also need to reconcile how relationships and hierarchies are selected. I don't necessarily want to go down a graphql path. A simple rubric might be that nodes that match the filter will also always include all of their parent nodes (even if parent nodes don't match a criteria; or possibly unless parent nodes don't match a specific filter criteria related to traversal). We might need a separate "where clause" (ie filter) and "select/include clause" (ie what to include from the matching objects and across relationships).

Example use cases to be fleshed out further, but off the top of my head:

• High level selected opportunities (no solutions or experiments) - select opportunity where ( wondering | exploring ) & opportunity (all layers above implied)
• Detail view of a goal - select goal where "Increase MAU" (title match) include goal to experiment
• All nodes reachable by parent links up and/or child links down from a given node

[mindsocket]

Regarding filtering - since we already use JSONata for expressions throughout the codebase, it would make sense to explore using JSONata for filtering as well. This would provide consistency and could leverage our existing infrastructure. Views could potentially be saved in config or schema files (to be decided later) as a convenience feature.

[mindsocket]

@claude research options for filtering from various angles (prior art, existing libraries, common patterns etc) based on the current state of this project and the thinking so far on this issue. Be sure to include assumptions or open questions that would need to be considered before deciding, and recommendations.

Respond with your findings here. Don't proceed if you can't access web resources needed to properly research.

[claude]

Filtering Research for OST Tools

Executive Summary

Based on codebase analysis, JSONata is the recommended approach for implementing filtering in ost-tools. It's already integrated for rule validation, well-suited for tree/graph structures, and provides powerful query capabilities.

Proposed CLI:

ost-tools validate ProductX --filter "resolvedType='solution' and status='active'"
ost-tools dump ProductX --filter "resolvedParentTitle='Increase user retention'"

Implementation effort: ~1-2 days for core --filter flag functionality

---

Current Filtering Landscape

Existing JSONata Integration

JSONata is already used in the codebase for rule validation (see src/schema/evaluate-rule.ts):

Evaluation Context (EvalContext):

• nodes - All nodes in the space
• $ / current - Current node being evaluated
• parent - First parent node
• parents - All parent nodes

Example Rules from Production (schemas/_ost_strict.json):

// Count nodes by type and status
"$count(nodes[resolvedType='outcome' and status='active']) <= 1"

// Parent-child relationships
"$count(nodes[resolvedParentTitle=$.current.title and resolvedType='solution']) >= 3"

// Status propagation
"current.status != 'active' or $exists(parent) = false or parent.status = 'active'"

JSONata Capabilities for OST Filtering

Use Case	JSONata Expression
Filter by type	resolvedType='opportunity'
Filter by status	status='active'
Parent relationship	resolvedParentTitle='Some Parent'
Multiple conditions	resolvedType='solution' and status='active'
Negation	not(status='archived')
Aggregations	$count(nodes[status='active'])
Custom fields	customField.value > 5

---

Proposed Implementation

Phase 1: CLI Flag (Priority)

Add --filter flag to existing commands:

// src/index.ts
program
  .command('validate <space>')
  .option('--filter <expr>', 'JSONata filter expression')
  .action(async (space, options) => {
    const { diagnostics } = await validate(space, {
      filter: options.filter
    });
  });

Implementation in src/validate.ts:

export async function validate(spaceName: string, options: { filter?: string }): Promise<ValidationResult> {
  const context = await loadSpace(spaceName);
  let nodes = context.nodes;
  
  if (options.filter) {
    const expression = jsonata(options.filter);
    const filteredResults = await expression.evaluate({
      nodes: context.nodes,
      current: null, // No current node in this context
    });
    nodes = filteredResults || [];
  }
  
  // Continue with validation on filtered nodes...
}

Phase 2: Interactive Filter Builder (Future)

• Auto-complete for field names
• Filter templates for common patterns
• Visual filter composition

---

Design Considerations

Assumptions

1. Tree-aware filtering: Users want to filter by tree position (parent/child relationships)
2. Status-based filtering: Active/archived/draft filtering is common
3. Type-based filtering: Filter by outcome/opportunity/solution/etc.
4. Aggregation queries: Count nodes matching criteria

Open Questions

1. Output format for dump command: Should filtered output preserve tree structure or return flat list?
2. Filter validation: Should invalid filter expressions fail fast or warn?
3. Performance: For large spaces (1000+ nodes), should filters be pre-compiled?
4. Filter persistence: Should users be able to save named filters in config?

Alternatives Considered

Approach	Pros	Cons	Verdict
JSONata	Already integrated, powerful, tree-aware	Learning curve for users	✅ Recommended
jq	Familiar to many, fast	Not tree-aware, harder parent queries	❌ Not suitable
JSONPath	Simple, standard	Limited expressiveness	❌ Too limited
Custom DSL	Full control	Maintenance burden, re-inventing wheel	❌ YAGNI

---

Prior Art & Patterns (Pending Web Research)

Still investigating:

• npm packages for tree/graph filtering
• GitHub/Gmail/Todoist filter syntax patterns
• CLI precedents for tree-structured data filtering
• Query language patterns for hierarchical data

---

Recommendation

Proceed with JSONata-based filtering for the following reasons:

1. ✅ Zero new dependencies - already in codebase
2. ✅ Tree-aware - can query parent/child relationships naturally
3. ✅ Powerful - supports aggregations, functions, composition
4. ✅ Validated - proven in production for rule checking
5. ✅ Extensible - can add custom functions if needed

Implementation Priority

1. Sprint 1: Add --filter to validate command (1 day)
2. Sprint 2: Add --filter to dump command (0.5 day)
3. Sprint 3: Add --filter to other commands as needed (0.5 day)
4. Future: Interactive filter builder, saved filters, etc.

---

Research in progress - awaiting web search access for comprehensive analysis of prior art, npm packages, and UX patterns from similar tools.

[mindsocket]

Follow-up research: tree traversal and graph structure

Picking up from the earlier JSONata recommendation — it's sound as a direction but the previous analysis didn't fully work through what tree-based queries require. This comment digs into that specifically.

---

Two separate problems

The issue description actually contains two distinct concerns that need different solutions:

1. Filter predicate — which nodes match (e.g. "solutions whose grandparent goal has status: active")
2. Result shape / inclusion — what to include alongside matches (e.g. "also include the parent opportunity even if it doesn't independently match")

These need different mechanisms. Mixing them into one expression makes the language harder to use.

---

The critical gap in the current representation

flattenNode() currently exposes resolvedParentTitle and resolvedParentTitles — one hop up. Multi-hop queries (grandparent, any ancestor) require cross-node lookups in JSONata.

In theory JSONata can do this — but with nested $map/$filter lambdas that must explicitly capture outer variables (because $ is shadowed inside nested predicates). For example, "solutions where grandparent goal has status: active" would look something like:

$filter(nodes[resolvedType='solution'], function($sol) {
  $exists(
    $filter(
      nodes[resolvedType='opportunity' and title=$sol.resolvedParentTitle],
      function($opp) {
        $exists(nodes[resolvedType='goal' and status='active' and title=$opp.resolvedParentTitle])
      }
    )
  )
})

That's not user-hostile for schema rule authors, but it's verbose and brittle (hardcodes two hops). It's also not the right model to expose for filtering — JSONata's strength is JSON projection, not graph walking.

---

Two paths forward for tree queries

Option A: Augmented flat nodes (ancestors array)

Pre-compute ancestry chains in flattenNode(). Add an ancestors array to each node, ordered from immediate parent to root:

{
  ...schemaData,
  resolvedType,
  resolvedParentTitle,   // existing — first parent
  resolvedParentTitles,  // existing — all parents
  ancestors: [           // NEW — ordered parent → grandparent → … → root
    { title: 'Opportunity X', resolvedType: 'opportunity', status: 'active' },
    { title: 'Goal A', resolvedType: 'goal', status: 'active' },
  ]
}

Then the grandparent query becomes:

nodes[resolvedType='solution' and ancestors[1].status='active']

Or more robustly (any depth):

nodes[resolvedType='solution' and $exists(ancestors[resolvedType='goal' and status='active'])]

This keeps JSONata as the engine and makes multi-hop queries clean. The tradeoff: ancestry must be computed at evaluation time, and in a DAG (multiple parents), ancestors would need to be an array-of-arrays or flattened with deduplication.

Option B: Build an actual graph structure

Pass a richer structure to JSONata rather than a flat array. For a tree (or forest), a nested representation:

{
  "roots": [
    {
      "title": "Goal A", "resolvedType": "goal", "status": "active",
      "children": [
        {
          "title": "Opportunity X", "resolvedType": "opportunity",
          "children": [
            { "title": "Solution Y", "resolvedType": "solution", ... }
          ]
        }
      ]
    }
  ],
  "nodes": [ /* flat array kept for global lookups */ ]
}

JSONata's ** (recursive descent) operator can then walk the tree:

roots.**.children[resolvedType='solution']

The problem: this codebase's hierarchy is a DAG (a node can have multiple parents), so a purely nested representation requires duplication or is impossible for multi-parent nodes. A nested structure works cleanly only for strict trees.

Recommendation: Option A with a depth-limited ancestors array. It avoids duplication issues with multi-parent nodes and keeps the flat nodes[] array as the primary query target. Option B could work as a complementary structure for display/rendering but not as the canonical query model.

---

The result shape / inclusion problem

This is where the tree structure matters most — and it's a post-filter operation, not part of the filter predicate. Patterns from the issue:

• "include parent nodes of matched nodes" → walk resolvedParents from each match, add to result set
• "include subtree from match down to depth N" → walk children map (already built by classifyNodes in graph-helpers.ts) from each match
• "top layers only" → depth limit on hierarchy levels

The existing classifyNodes / buildHierarchyNodeSet infrastructure already provides what's needed for this. The design question is how to expose it at the CLI level — probably something like:

--filter "resolvedType='solution'"  # which nodes match
--include ancestors                 # also include parents of matches
--include subtree:2                 # also include up to 2 levels of children

Or a more compact inline form TBD.

---

Prior art summary (web research)

• XPath axes (ancestor::, descendant::) are the best semantic precedent for what the operations mean. //solution[ancestor::goal[@status='active']] directly expresses the grandparent query. The axis vocabulary is precise but unfamiliar to non-technical users.
• CSS combinators (>, space, ~) are the most learnable for parent/child relationships. The :has() pseudo-class (CSS4) partially addresses ancestor queries but still can't express "select B whose ancestor has property X" cleanly.
• Cypher ([:PARENT_OF*] variable-length paths) is the clearest way to express "ancestor at any depth". (y:Goal {status:'active'})-[:PARENT_OF*]->(x:Solution) is readable and explicit about direction.
• No existing JS/TS library was found that provides a declarative query language for flat node arrays with ancestor-aware filtering. Most tools work on nested tree objects (d3-hierarchy) or are full graph databases. This is a genuine gap — building something here would be novel.
• AQL (ArangoDB) uses LET ancestors = (FOR a IN 1..10 INBOUND x edge RETURN a) — closest in concept to what we'd compute, but verbose. The pattern of "capture ancestors into a variable, then filter on the array" maps well to the augmented flat nodes approach.

Practical synthesis: a readable user-facing filter language might borrow:

• type=solution for simple property predicates (Todoist-style)
• [ancestor: type=goal, status=active] or ancestor(type=goal, status=active) for ancestor predicates (XPath concept, function-call syntax)
• Separate include: parents / include: subtree modifiers for result shape (no query language handles this elegantly inline)

---

Open questions before designing further

1. Should the user-facing filter syntax compile to JSONata, or should JSONata only power schema rules while filtering uses a different (simpler) evaluator?
2. How should multi-parent nodes be handled in ancestry chains — do we include all paths, or just hierarchy-sourced parents?
3. Is the "include" clause a CLI flag, a separate part of a filter expression, or out of scope for an initial implementation?
4. What's the target user for --filter? If schema authors (power users), JSONata with augmented nodes may be fine. If product managers in a future UI, a simpler syntax is worth the investment.

[mindsocket]

Design decisions and theoretical spec (current state)

Summarising the thinking so far into a more complete picture. This supersedes the earlier research comments.

---

Two-expression model

Filtering has two distinct concerns handled by separate expressions:

1. Filter expression — determines which nodes are in the matched set. Evaluated per node. Returns boolean.

2. Include expression — expands the result set by walking the graph from matched nodes. Determines what gets pulled in alongside matches (ancestors, descendants, relationship-linked nodes). Separate mini-language, not JSONata.

These map roughly to SQL's WHERE (filter) and the implicit JOIN/INCLUDE shape of a result.

---

Node augmentation

To make multi-hop tree queries expressible, each node in the evaluation context will be augmented with:

ancestors: [{ title, resolvedType, status, ..., _field, _source, _selfRef }, ...]
descendants: [{ title, resolvedType, status, ..., _field, _source, _selfRef }, ...]

• Flat arrays (not nested, not per-path) — duplicates removed where multiple paths exist (DAG case)
• Each entry merges the node's data with edge metadata from resolvedParents (_field, _source: 'hierarchy'|'relationship', _selfRef)
• Ordered nearest-first (ancestors: parent before grandparent; descendants: children before grandchildren)
• Self-referential hops are traversed through — so a solution with two self-referential solution ancestors before reaching an opportunity would have all three in ancestors[]

This makes ancestor queries expressible without cross-node lookups:

resolvedType = 'solution' and $exists(ancestors[resolvedType = 'goal' and status = 'active'])

resolvedType = 'solution' and $exists(ancestors[_source = 'hierarchy' and resolvedType = 'goal'])

---

Filter expression language: JSONata

JSONata is confirmed as the filter language (user-facing). JMESPath was considered and rejected — it cannot express aggregations inside filter predicates (e.g. length(ancestors[?status=='active']) > 0 is not valid in standard JMESPath) and does not support custom function binding. JSONata handles both.

The augmented ancestors/descendants arrays mean most realistic filter expressions don't require advanced JSONata features:

resolvedType = 'solution' and status = 'active'

resolvedType = 'opportunity' and $exists(ancestors[resolvedType = 'goal' and status = 'active'])

resolvedType = 'solution' and $count(descendants[resolvedType = 'experiment']) >= 1

---

Include expression language (sketch)

A separate mini-language for the include clause. Not JSONata. Combines:

Convenience functions:

• ancestors — all ancestors of matched nodes
• descendants — all descendants of matched nodes
• siblings — nodes sharing a parent with matched nodes
• relationships — all relationship-linked nodes (non-hierarchy edges)

Type names — include nodes of a specific type reachable from matches:

• opportunity — direct parents of type opportunity
• problem_statement — relationship children of that type

Hierarchy level ranges — a slice of the hierarchy:

• opportunity..goal — from opportunity level up to goal level
• solution..experiment — matched node's subtree down to experiment

Relationship specs — by child type name (relationships have no standalone name, only parent + type):

• problem_statement, assumption, risk

Include and exclude directions — both supported:

• include: ancestors / exclude: descendants[resolvedType='experiment']

The exact grammar is not yet finalised. The goal is something compact enough for a CLI flag and storable in config/schema files — same format in both contexts, SQL-query-level familiarity.

---

Storage / invocation

Both expressions are plain strings. No distinction in richness between ad-hoc CLI use and stored views:

ost-tools show myspace --filter "resolvedType='solution' and status='active'" --include "ancestors"

Stored views (in config or schema, design TBD) would use the same strings.

---

Open questions

1. Exact grammar for the include expression (sketch above, not finalised)
2. Edge metadata field naming convention in augmented nodes (_field/_source prefix or something else)
3. How stored views are declared and named (config file, schema $metadata, or separate?)
4. Whether descendants traversal follows hierarchy edges only or also relationship edges (or is this controlled per query via _source filtering?)

[mindsocket]

Relationship reference syntax in include expressions

Relationships have no standalone name — they're identified by parent + type + field. The include expression uses progressive specification to keep common cases terse:

Form	When to use
child_type	Child type is unambiguous across all relationships in the schema
parent_type:child_type	Child type alone is ambiguous (appears under multiple parents)
parent_type:field:child_type	Same parent+child pair exists with multiple fields

The third form is needed when a parent type has two relationships to the same child type via different fields. Example:

{ "parent": "Activities", "type": "Data", "field": "data_required", ... },
{ "parent": "Activities", "type": "Data", "field": "data_produced", ... }

These would be referenced as Activities:data_required:Data and Activities:data_produced:Data respectively. Activities:Data alone would be an error (ambiguous).

[mindsocket]

Addressed in #66