feat: Implement Batch Evaluator by adnanrhussain · Pull Request #16 · learning-commons-org/evaluators

adnanrhussain · 2026-02-09T17:47:49Z

Summary

A CLI tool that processes CSV files containing educational text through multiple AI evaluators (vocabulary, sentence structure, grade-level appropriateness) in parallel, outputting results in CSV, JSON, and interactive HTML formats with progress tracking and graceful shutdown support.

Example of Report HTML

Documentation

See sdks/typescript/src/batch/README.md

Testing

CI workflow should build successfully

Copilot

Pull request overview

Adds a new TypeScript “batch” feature: an interactive CLI and supporting runtime APIs for evaluating many CSV rows across multiple evaluators in parallel, producing CSV/JSON/HTML outputs.

Changes:

Builds and ships a new CLI entrypoint (src/batch/index.ts) plus batch runtime modules (evaluator, formatters, progress, types).
Updates build/package wiring to include the batch entry in tsup and expose a new evaluators-batch binary.
Adds unit + integration tests and a CSV fixture for batch parsing/formatting.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
`sdks/typescript/tsup.config.ts`	Adds `src/batch/index.ts` as a build entrypoint so the CLI is emitted into `dist/`.
`sdks/typescript/package.json`	Adds `bin` entry + new deps for CSV parsing and interactive prompts.
`sdks/typescript/package-lock.json`	Locks new dependencies introduced for batch functionality.
`sdks/typescript/src/utils/prompts.ts`	Adjusts prompt directory resolution to support the batch build output layout.
`sdks/typescript/src/evaluators/base.ts`	Introduces an abstract `evaluate(text, grade)` method on the base class.
`sdks/typescript/src/evaluators/text-complexity.ts`	Removes an unused import.
`sdks/typescript/src/batch/types.ts`	Defines batch input/result/output/config types.
`sdks/typescript/src/batch/evaluator.ts`	Implements concurrency-controlled batch execution + cancellation + summary aggregation.
`sdks/typescript/src/batch/formatters.ts`	Implements CSV/JSON/HTML output formatting for batch results.
`sdks/typescript/src/batch/progress.ts`	Implements terminal progress rendering and summary display.
`sdks/typescript/src/batch/index.ts`	Implements the interactive CLI flow (CSV selection, evaluator selection, keys, outputs, shutdown).
`sdks/typescript/src/batch/README.md`	Adds end-user documentation for running the CLI and interpreting outputs.
`sdks/typescript/tests/unit/batch/*.test.ts`	Adds unit tests for formatters and CSV parsing helpers and limit math.
`sdks/typescript/tests/integration/batch.integration.test.ts`	Adds an opt-in integration test that exercises the batch evaluator with real API keys.
`sdks/typescript/tests/fixtures/batch-test.csv`	Adds a fixture CSV with column-name variants to validate robust parsing.

Files not reviewed (1)

sdks/typescript/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sdks/typescript/tests/unit/batch/csv-parsing.test.ts

sdks/typescript/package.json

sdks/typescript/src/batch/README.md

sdks/typescript/package.json

sdks/typescript/src/batch/index.ts

sdks/typescript/src/batch/progress.ts

sdks/typescript/src/evaluators/base.ts

sdks/typescript/src/batch/evaluator.ts

sdks/typescript/tests/unit/batch/formatters.test.ts

sdks/typescript/src/batch/index.ts

Copilot

Pull request overview

Copilot reviewed 18 out of 20 changed files in this pull request and generated 6 comments.

Files not reviewed (1)

sdks/typescript/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

sdks/typescript/src/batch/README.md

sdks/typescript/tests/integration/batch.integration.test.ts

sdks/typescript/src/batch/cli.ts

sdks/typescript/src/batch/README.md

codecov · 2026-03-17T00:04:52Z

Codecov Report

❌ Patch coverage is 78.94737% with 60 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
sdks/typescript/src/batch/evaluator.ts	26.25%	59 Missing ⚠️
sdks/typescript/src/batch/formatters.ts	99.42%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

adnanrhussain · 2026-03-17T00:41:09Z

sdks/typescript/src/evaluators/base.ts

        if (a === 'K') return -1;
        if (b === 'K') return 1;
-        return parseInt(a) - parseInt(b);
+        return parseInt(a, 10) - parseInt(b, 10);


Unrelated to the scope of this particular PR

czi-fsisenda

Very nice PR! Love the structure. Easy to follow for such a large feature.
Mostly nits and some suggestions for future iterations.
One blocking issue: I think the integration tests should run against all available evals and should fail instead of skip for missing api keys.

czi-fsisenda · 2026-03-19T02:20:35Z

sdks/typescript/src/batch/report-template.html

Rubber stamp. Already reviewed.

sdks/typescript/src/batch/cli.ts

czi-fsisenda · 2026-03-19T03:00:12Z

sdks/typescript/src/batch/types.ts

+/**
+ * Input row from CSV
+ */
+export interface BatchInput {


nit: BatchRow

czi-fsisenda · 2026-03-19T03:02:26Z

sdks/typescript/src/batch/types.ts

+/**
+ * Individual evaluation task
+ */
+export interface BatchTask {
+  text: string;
+  grade: string;
+  evaluatorId: string;
+  rowIndex: number;
+}


[P1] generic input vs text and grade

sdks/typescript/tests/integration/batch.integration.test.ts

czi-fsisenda · 2026-03-19T03:21:03Z

sdks/typescript/src/batch/formatters.ts

+  totalInputRows: number;
+}
+
+export function formatAsHTML(output: BatchOutput, meta: ReportMeta): string {


[P2] I wonder if we can share a common template once the Playground has batch support too.

adnanrhussain · 2026-03-19T05:21:28Z

@czi-fsisenda - Thank you for your review. Addressed feedback. Please review the last commit - b1dbb69

czi-fsisenda

🚀 Nice! Even got a P1 in!

adnanrhussain requested a review from Copilot February 19, 2026 22:45

Copilot started reviewing on behalf of adnanrhussain February 19, 2026 22:45 View session

Copilot AI reviewed Feb 19, 2026

View reviewed changes

adnanrhussain force-pushed the ahussain/sdk_6_composite_evals branch from 173a014 to 13325f1 Compare March 3, 2026 17:37

adnanrhussain force-pushed the ahussain/sdk_7_batch branch from 1e50c35 to a412e0b Compare March 5, 2026 04:53

Base automatically changed from ahussain/sdk_6_composite_evals to ahussain/sdk_typescript March 5, 2026 23:07

adnanrhussain force-pushed the ahussain/sdk_7_batch branch from 44c78cd to e6621b5 Compare March 12, 2026 17:24

Base automatically changed from ahussain/sdk_typescript to main March 13, 2026 19:40

adnanrhussain force-pushed the ahussain/sdk_7_batch branch from e6621b5 to d501654 Compare March 17, 2026 00:01

adnanrhussain requested a review from Copilot March 17, 2026 00:01

Copilot started reviewing on behalf of adnanrhussain March 17, 2026 00:02 View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

adnanrhussain force-pushed the ahussain/sdk_7_batch branch from d501654 to 2b1311a Compare March 17, 2026 00:36

adnanrhussain commented Mar 17, 2026

View reviewed changes

adnanrhussain requested a review from czi-fsisenda March 17, 2026 05:54

adnanrhussain marked this pull request as ready for review March 17, 2026 05:54

adnanrhussain force-pushed the ahussain/sdk_7_batch branch from 5a5816d to c77b424 Compare March 18, 2026 18:21

czi-fsisenda requested changes Mar 19, 2026

View reviewed changes

adnanrhussain requested a review from czi-fsisenda March 19, 2026 05:21

czi-fsisenda approved these changes Mar 19, 2026

View reviewed changes

adnanrhussain added 8 commits March 23, 2026 16:12

feat: Implement Batch Evaluator

55f9b28

Update report template

50e3de1

load test and limit to 50

ba6950b

fix limit in tests

039c42f

template design updates

bdcf4f6

add SMK to Batch Evaluator

51b979c

re: feedback

dd515e5

add Conventionality to Batch Evaluator

e57fa34

adnanrhussain added 2 commits March 23, 2026 16:46

update template + bump version

88a16e2

documentation cleanup

a307a42

adnanrhussain force-pushed the ahussain/sdk_7_batch branch from b1dbb69 to a307a42 Compare March 23, 2026 23:49

adnanrhussain merged commit a0c78ab into main Mar 23, 2026
11 checks passed

adnanrhussain deleted the ahussain/sdk_7_batch branch March 23, 2026 23:53

czi-github-helper bot mentioned this pull request Mar 23, 2026

chore(main): release 1.1.0 #3

Open

Conversation

adnanrhussain commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example of Report HTML

Documentation

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

adnanrhussain Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

czi-fsisenda left a comment

Choose a reason for hiding this comment

Uh oh!

czi-fsisenda Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

czi-fsisenda Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

czi-fsisenda Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

czi-fsisenda Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

adnanrhussain commented Mar 19, 2026

Uh oh!

czi-fsisenda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adnanrhussain commented Feb 9, 2026 •

edited

Loading

codecov bot commented Mar 17, 2026 •

edited

Loading