chore(eval): resync evaluator type schemas with Python source#1664
Draft
ajay-kesavan wants to merge 1 commit into
Draft
chore(eval): resync evaluator type schemas with Python source#1664ajay-kesavan wants to merge 1 commit into
ajay-kesavan wants to merge 1 commit into
Conversation
Refreshes the 11 evaluator schemas in evaluators_types/ that drifted from their Python config classes since the last regeneration. Pure output of `python -m uipath.eval.evaluators_types.generate_types`, no hand edits. Notable additions surfaced by the regenerator: - Output evaluators (Contains, ExactMatch, JsonSimilarity, LLMJudge*) gain line_by_line_evaluator and line_delimiter, added to OutputEvaluatorConfig in #1481. - All evaluators gain a populated justificationSchema where previously empty. - ToolCall evaluators gain target_output_key documentation. Split out from #1663 (which now ships only the two new classification schemas) so this snapshot refresh can be reviewed independently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 tasks
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refreshes the 11 evaluator schemas in
packages/uipath/src/uipath/eval/evaluators_types/that had drifted from their Python config classes. Pure output ofpython -m uipath.eval.evaluators_types.generate_types, no hand edits.What changed
Notable fields the regenerator surfaced:
line_by_line_evaluatorandline_delimiter— added toOutputEvaluatorConfigin feat: add support for line-by-line evaluation #1481 but never propagated to the JSON snapshot.justificationSchema(previously emitted as empty{}).target_output_keydocumentation strings.Why split this out
Originally bundled with the classification-evaluator schemas in #1663, but split into its own PR so reviewers don't have to mentally separate "new feature" from "snapshot refresh of unrelated config".
Test plan
OutputEvaluatorConfigor evaluator-specific config classes inpackages/uipath/src/uipath/eval/evaluators/python -m uipath.eval.evaluators_types.generate_typeslocally and confirmgit statusis cleanRelated
line_by_line_evaluatorfields without regenerating the JSONCo-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
🤖 Generated with Claude Code