Jobs to run of Apps Codegen bulk gen + Evals#4103

Open

keugenek wants to merge 17 commits intomainfrom

exprimental/mcp-evals-nightly

Commits on Dec 5, 2025

Commits on Dec 9, 2025

Fix app generation: use LiteLLM backend + fix UC Volume symlinks

keugenek
and
factory-droid[bot]
committed
Fix eval job CLI parameter parsing

keugenek
and
factory-droid[bot]
committed

Commits on Dec 10, 2025

Commits on Dec 11, 2025

Required to bypass proc mount restrictions and AppArmor.
keugenek
committed
Refactor get_prompts function to use external import for prompt retrieval, simplifying the code and enhancing maintainability.
keugenek
committed
Update README.md for Apps-MCP Evals: Enhance documentation to clarify job structure, prerequisites, and configuration details. Introduce Generation and Evaluation jobs, update quick start commands,…
keugenek
committed
Merge branch 'main' into exprimental/mcp-evals-nightly
keugenek
authored
Refactor eval setup and runner: Remove Docker installation and management from setup_eval.sh and run_evals.py, simplifying the evaluation process. Update Node.js installation comment for clarity an…
keugenek
committed
Add local evaluation functionality: Implement run_local_evaluation function to execute app evaluations without Docker, enhancing the evaluation process. Update main function to utilize local mode a…
keugenek
committed