Replace custom validator with skill-validator CLI

[marc0olo]

Summary

Replace the custom scripts/validate-skills.js with the skill-validator CLI — the reference validator for the Agent Skills spec. This gives us stronger validation (link checking, content analysis, contamination detection), better PR visibility, and a local LLM scoring workflow for contributors.

Motivation

The current JS validator covers basic frontmatter and section checks. skill-validator is a superset that additionally validates:

• External links — catches dead URLs in skills
• Content quality — information density, code ratio, instruction specificity
• Cross-language contamination — detects code examples that could cause incorrect generation in other language contexts
• Code fence integrity, token limits, orphan files — structural checks beyond what we have today

It also has first-class CI support with GitHub Actions annotations and markdown output.

Dependencies

• Claude CLI provider: The local LLM scoring workflow for contributors depends on the claude-cli provider being merged upstream: feat: add claude-cli provider for LLM scoring without API keys agent-ecosystem/skill-validator#43. This is only needed for the contributor docs / local scoring recommendation (step 6), not for CI validation itself.

Implementation Steps

1. Install skill-validator in CI

Add a step in _checks.yml to install the skill-validator binary (via go install or downloading a release binary).

2. Replace the validation command

Replace node scripts/validate-skills.js with:

skill-validator check skills/ -o markdown

Run without --strict initially — errors block, warnings are informational. The existing skills were written against the lighter validator and will likely produce new warnings that shouldn't block PRs until they're addressed.

3. Extract the evals existence check

The current validator checks for evaluations/<skill-name>.json — this is project-specific and not covered by skill-validator. Extract it into a small standalone script (scripts/check-evals.js) and run it alongside skill-validator.

4. Update npm scripts

Update package.json so npm run validate (which gates dev and build) runs both:

"validate": "skill-validator check skills/ && node scripts/check-evals.js"

5. Add PR reporting

• Use --emit-annotations to surface warnings as inline annotations on the PR diff
• Write the markdown output to $GITHUB_STEP_SUMMARY for the workflow run
• Add a sticky PR comment (via marocchino/sticky-pull-request-comment or similar) that updates in-place on each push — keeps the latest validation report visible at the top of the PR conversation without comment spam

6. Update contributor docs

CONTRIBUTING.md:

• Update Step 3 (Validate) to reference skill-validator check instead of just npm run validate
• Add a new step between validation and evals: recommend running LLM scoring locally before submitting a PR:

  skill-validator score evaluate --provider claude-cli skills/<skill-name>

  Frame as a quality check — low novelty scores indicate the skill may restate common knowledge
• Update Step 6 (Submit a PR) to suggest including scoring output in the PR description
• Remove the reference to KNOWN_CATEGORIES in validate-skills.js from the "To add a new category" section (that file goes away)

CLAUDE.md:

• Update the build commands section to include skill-validator
• Update the workflow section to reflect the new validation tooling

7. Add .score_cache to .gitignore

LLM scoring caches results in .score_cache/ directories inside each skill directory. These should not be committed — add .score_cache to the project .gitignore.

8. Migrate to --strict (follow-up)

Once existing skills are cleaned up and no longer produce warnings, flip to --strict so warnings also block PRs.

Out of Scope

• LLM scoring in CI — non-deterministic, slow, and requires API keys or CLI auth. Better suited for local use by contributors before submitting PRs.