Skip to content

Eval-2: eligibility completeness and childcare subsidy patterns#121

Merged
hua7450 merged 3 commits intoPolicyEngine:mainfrom
hua7450:eval-2-skill-improvements
Mar 9, 2026
Merged

Eval-2: eligibility completeness and childcare subsidy patterns#121
hua7450 merged 3 commits intoPolicyEngine:mainfrom
hua7450:eval-2-skill-improvements

Conversation

@hua7450
Copy link
Collaborator

@hua7450 hua7450 commented Mar 9, 2026

Summary

Second RI CCAP eval found the rules-engineer agent was dropping eligibility requirements despite them being in the spec and reference implementations. Also missing the standard childcare subsidy calculation pattern.

Fixes gaps identified in eval-2 (follow-up to #120).

Changes

Rules-engineer agent (agents/country-models/rules-engineer.md)

  • Step 1: Cross-reference every eligibility variable type from reference implementations against target program
  • Step 4: Renamed to two-pass completeness check — first verifying every spec requirement has a parameter+variable, then checking no parameters are orphaned

Code-organization skill

  • Added {prefix}_activity_eligible to standard eligibility variable components
  • Added parameter folder organization guidance (don't put rate dimension boundaries in eligibility folder)

Variable-patterns skill

  • Added activity_eligible to parameter-to-variable mapping table
  • Added childcare subsidy benefit calculation pattern using pre_subsidy_childcare_expenses
  • Added more common implementation failure modes

Eval-2 findings (12/12 eval-1 issues passed, 4 new issues found)

Issue Root Cause Fix
Missing work requirement variable (parameter created but orphaned) Agent didn't implement all spec requirements Two-pass completeness check in agent
Missing asset/resource eligibility Not in spec + agent didn't cross-reference MA CCFA Agent cross-references reference eligibility types
Missing immigration eligibility In spec but agent skipped it Two-pass completeness check in agent
Subsidy calc ignores actual expenses Agent didn't study MA CCFA benefit formula Childcare subsidy pattern added to skill
Parameter folder mixing concerns No guidance on folder purpose Folder organization guidance in code-org skill

🤖 Generated with Claude Code

hua7450 and others added 3 commits March 9, 2026 02:10
…tterns

Second RI CCAP eval revealed the rules-engineer agent was missing eligibility
variables (work requirements, assets, immigration) despite them being in the
spec and reference implementations. Also missing childcare subsidy pattern
using pre_subsidy_childcare_expenses.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Person-level variables like reimbursement rates should use
defined_for with the person-level eligibility variable, not just
the state code, to avoid unnecessary computation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When using select() with Enum variables, list all values explicitly
and set default to match the Enum's default_value for consistency.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@hua7450 hua7450 merged commit ec21243 into PolicyEngine:main Mar 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant