Snapshot record then reply testing#740
Merged
Merged
Conversation
mateusz834
force-pushed
the
dev/MP/snapshots
branch
16 times, most recently
from
1198d47 to
d029c1d
Compare
mateusz834
force-pushed
the
dev/MP/snapshots
branch
from
d029c1d to
7f769bb
Compare
mateusz834
requested review from
Ickerday and
szykol
and removed request for
szykol
Contributor
splunk-dtaborski
left a comment
splunk-dtaborski
left a comment
There was a problem hiding this comment.
I really like the solution.
szykol
reviewed
Apr 24, 2026
szykol
reviewed
Apr 24, 2026
szykol
reviewed
Apr 24, 2026
szykol
reviewed
Apr 24, 2026
szykol
reviewed
Apr 24, 2026
szykol
reviewed
Apr 24, 2026
mateusz834
force-pushed
the
dev/MP/snapshots
branch
2 times, most recently
from
32aadd8 to
6aa74cd
Compare
This change implements snapshot, record then reply testing for all integration tests. Re-enables recently skipped (falky) tests: - test_agent_understands_other_agents (snapshot was edited manually) - test_supervisor_resumes_subagent_thread_across_invocations - test_supervisor_resumes_subagent_thread_across_invocations_structured Introduces a deterministic thread_id mock generator, such that snapshots are deterministic, for: - test_supervisor_resumes_subagent_thread_across_invocations - test_supervisor_resumes_subagent_thread_across_invocations_structured Modified test_tool_execution_service_access using tool middleware, to make the test deterministic.
mateusz834
force-pushed
the
dev/MP/snapshots
branch
from
6aa74cd to
cd1a2ee
Compare
szykol
approved these changes
Apr 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change implements snapshot, record then reply testing for all integration tests.
Re-enables recently skipped (flaky) tests:
test_agent_understands_other_agents(snapshot was edited manually)test_supervisor_resumes_subagent_thread_across_invocationstest_supervisor_resumes_subagent_thread_across_invocations_structuredIntroduces a deterministic thread_id mock generator, such that snapshots are deterministic, for:
test_supervisor_resumes_subagent_thread_across_invocationstest_supervisor_resumes_subagent_thread_across_invocations_structuredModified
test_tool_execution_service_accessusing tool middleware, to make the test deterministic.E2E tests still call the LLMs directly.