Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
There was a problem hiding this comment.
Pull request overview
Removes the unused metrics configuration section from the benchmark configuration schema and updates templates/examples/tests accordingly so configs no longer include or validate that field.
Changes:
- Removed the
MetricsPydantic model andmetricsfield fromBenchmarkConfig. - Updated config templates and example YAMLs to drop the
metrics:section. - Updated unit tests to stop including/validating
metricsin YAML/schema tests.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/config/test_yaml_loader.py | Removes metrics from a valid YAML config used in loader tests. |
| tests/unit/config/test_schema.py | Removes Metrics import and the Metrics unit tests. |
| src/inference_endpoint/config/templates/submission_template.yaml | Drops metrics section from submission template. |
| src/inference_endpoint/config/templates/online_template_full.yaml | Drops metrics section from online full template. |
| src/inference_endpoint/config/templates/offline_template_full.yaml | Drops metrics section from offline full template. |
| src/inference_endpoint/config/templates/eval_template.yaml | Drops metrics section from eval template. |
| src/inference_endpoint/config/templates/concurrency_template_full.yaml | Drops metrics section from concurrency full template. |
| src/inference_endpoint/config/schema.py | Removes Metrics model and BenchmarkConfig.metrics field. |
| examples/08_Qwen3-VL-235B-A22B_Example/online_qwen3_vl_235b_a22b_shopify.yaml | Drops metrics section from example config. |
| examples/08_Qwen3-VL-235B-A22B_Example/offline_qwen3_vl_235b_a22b_shopify.yaml | Drops metrics section from example config. |
| examples/06_Llama2-70B_Example/online_llama2_70b_orca.yaml | Drops metrics section from example config. |
| examples/05_Llama3.1-8B_Example/online_llama3_8b_cnn.yaml | Drops metrics section from example config. |
| examples/05_Llama3.1-8B_Example/offline_llama3_8b_cnn.yaml | Drops metrics section from example config. |
| examples/04_GPTOSS120B_Example/vllm_gptoss_120b_example.yaml | Drops metrics section from example config. |
| examples/04_GPTOSS120B_Example/sglang_gptoss_120b_example.yaml | Drops metrics section from example config. |
| examples/04_GPTOSS120B_Example/gptoss_120b_example.yaml | Drops metrics section from example config. |
| examples/02_ServerBenchmarking/online_llama2_70b_cnn.yaml | Drops metrics section from example config. |
| examples/02_ServerBenchmarking/offline_llama3_8b_cnn.yaml | Drops metrics section from example config. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Code Review
This pull request removes the metrics collection configuration and associated logic from the codebase, including the Metrics model, its helper methods, and all references in YAML configuration templates and unit tests. I have no feedback to provide as the changes are straightforward and clean.
What does this PR do?
based on offline discussion, we want to drop metrics from schema (it was unused anyways).
Type of change
Related issues
Testing
Checklist