feat: add stream_validate() hook to Requirement (#900)

[planetf1]

Requirement PR

Description

• Link to Issue: Closes feat(core): add stream_validate() to Requirement #900

Adds async stream_validate(chunk, *, backend, ctx) -> PartialValidationResult to the base Requirement class as a per-chunk streaming validation hook. The default implementation returns PartialValidationResult("unknown"); subclasses override to inspect the current chunk (the semantic delta from the chunking strategy — one sentence, word, or paragraph per call) and signal "pass" or "fail" early. Stateful implementations maintain their own running state across calls (e.g. self._count += chunk.count("•")); the orchestrator (in the stacked PR #942) clones the requirement before each attempt so state does not bleed across retries.

Part of streaming epic #891. Builds on #924 (merged — PartialValidationResult) and #923 (merged — ChunkingStrategy).

Design decisions

Per the agreed Phase 1 spec:

• Method name: stream_validate (not avalidate) — different operation, different semantics
• Return type: PartialValidationResult (tri-state "pass" / "fail" / "unknown") — from feat(core): add PartialValidationResult with tri-state semantics #898 / feat(core): add PartialValidationResult with tri-state semantics #924
• Chunk semantics: chunk is a single complete chunk from the chunking strategy (the delta since the previous call), not the full accumulated output. This matches the feat: streaming validation — per-chunk requirement checking with early exit #891 epic's "one chunk at a time" contract and the feat(core): add stream_validate() to Requirement #900 spec.
• backend and ctx are keyword-only: prevents positional confusion and keeps future parameter additions non-breaking
• "pass" is informational: does not short-circuit the final validate() call in Phase 1
• No reset() method: state isolation is handled by orchestrator cloning (copy(req)) before each attempt

No LLM-as-a-Judge logic here — this is a pure hook for custom validation overrides.

Review feedback addressed

@jakelorocco flagged on first pass that an earlier iteration of this branch passed the accumulated output to stream_validate rather than a single chunk, which deviated from the spec. Commit 82bdd3a5 restores the spec-compliant behaviour:

• requirement.py docstring rewritten to describe chunk as the delta from the chunking strategy, with notes on ctx being incomplete during streaming and the MOT single-consumer constraint.
• test_stateful_subclass_accumulates_state rewritten: BulletCounter now accumulates on self._count across delta calls, dropping the earlier self._seen_len workaround that only made sense under accumulated semantics.

The corresponding orchestrator change (pass single chunk, not accumulated) is in the stacked PR #942.

Implementation Checklist

Base Class

• Extends appropriate base class:
  • Requirement - standard requirement

Validation Logic

• Hook method on base Requirement; no default validation_fn added (stream_validate is an override point, not a validation implementation)
• Returns PartialValidationResult with tri-state success and optional reason / score

Integration

• No change to mellea/stdlib/requirements/__init__.py needed — this PR modifies the base class, not a new requirement

Testing

• Tests added to test/core/test_stream_validate.py:
  • test_default_returns_unknown — base class always returns "unknown"
  • test_default_returns_partial_validation_result_instance — correct return type
  • test_stream_validate_is_coroutine — method is async
  • test_subclass_can_return_pass / test_subclass_can_return_fail — subclass overrides work
  • test_does_not_mutate_requirement / test_stream_validate_idempotent — base class is stateless
  • test_stateful_subclass_accumulates_state — delta-semantics bullet counter correctly accumulates on self
  • test_stateful_subclass_clone_isolation — copy() produces independent clones for orchestrator use
• New code has 100% coverage
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

• AI coding assistants used

[planetf1]

@jakelorocco how does this look for you - I have another stacked PR behind this one too. (I'll just go one level deep)

[jakelorocco]

@planetf1, I might've missed this in the original proposal, but passing the accumulated chunks to the requirements is different from what I thought the proposal was. Could you please elaborate more on the design choice?

[jakelorocco]

This worries me. @nrfulton, your initial proposal was to have the requirement only see new chunks. This would show all accumulated chunks so far.

This forces all streaming requirements to be stateful; all requirements must now keep track of what chunks they have processed. The alternative would be to only provide new chunks to requirements; then streaming validation would be stateless except when needed. Requirements can choose whether they need to store and process multiple chunks, or just check each chunk independently.

I guess the checking each chunk independently is unlikely to be helpful, so I can see why accumulating and forcing requirements to track their own progress doesn't actually add much complexity.

If so, I think we should actually pre-define functions to help with this (either as a new class of requirements or functions that implementors can draw on).

Also, if we are passing the accumulated chunk through, I almost think we should just pass in some point-in-time copy of the model output thunk. Ie one that doesn't get streamed the new chunks but has all the data fields from the point-in-time it was copied at.

[planetf1]

I would suggest reverting to the simplicity of the original proposal. The change was made incorrectly when reviewing the initial PR. The initial approach is simple - and requirements can maintain state if the need to work on accumulated chunks. If we need to support accumulated content generally we can consider it in a later phase.

Which works best will vary by use case. Per-chunk like works better for

• checkinf for forbidden words/phrases
• ensuring a paragraph or sentence is coherent
• structural checks - code fencing
• format validation

especially when the MoT manages the semantic chunking (later)

There will be cases where accumulation is better -- these are probably more complex checks - does a story line flow, are we taking the response in an unexpected direction, do we have a complete enough response

Importantly if we stick to per-chunk we could still implement this second approach - albeit not as cleanly.

So in summary - I'll revert to original -- but if you now think that's wrong we can adjust?

[jakelorocco]

lgtm; I think the single chunk approach is good. If we want to revert back to the accumulation, I don't think we are stuck with this approach yet.

[jakelorocco]

Clarification, the generation context has the uncomputed mot at this point, right? That's the reason for the warning?

[planetf1]

Yes - they need to rely on the chunk & anything they've accumulated themselves. mot_is_computed stays false until streaming ends so we can't really say for sure what's in it. Tried to ensure we capture the behaviour in the docstrings

I think any changes to this initial approach -- and making the mot more responsible -- is a later phase.

Thanks for approval!

I have a stacked PR which I'll get out asap too (may not be until tomorrow)