feat(report): enhanced error reporting with troubleshooting hints#68
feat(report): enhanced error reporting with troubleshooting hints#68ian-flores wants to merge 3 commits intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds richer failure reporting to VIP’s Quarto report by attaching pytest-bdd scenario metadata to test results, loading per-scenario troubleshooting hints from TOML, and rendering both hints and BDD steps in the report output (including a failures.json export for downstream automation).
Changes:
- Extend results JSON /
TestResultwith optionalscenario_titleandfeature_descriptioncaptured during pytest collection. - Add TOML-driven troubleshooting hint loader and integrate hint rendering +
failures.jsonexport in the report. - Introduce a lightweight
.featureparser to show BDD scenario steps in report details.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/troubleshooting.toml | Adds scenario-title keyed troubleshooting hints surfaced in the report. |
| src/vip/reporting.py | Adds optional scenario metadata fields + TOML troubleshooting loader. |
| src/vip/plugin.py | Stashes pytest-bdd scenario metadata and emits it into results JSON. |
| src/vip/gherkin.py | Adds a minimal Gherkin .feature parser for step extraction. |
| selftests/test_reporting.py | Adds coverage for new TestResult fields + load_troubleshooting(). |
| selftests/test_gherkin.py | Adds coverage for the new Gherkin parser. |
| report/index.qmd | Renders “what was tested”, troubleshooting hints, and exports failures.json. |
| report/details.qmd | Adds expandable per-test BDD step rendering via parsed .feature files. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
src/vip/reporting.py
Outdated
| ``docs_url``. Returns an empty dict if the file does not exist. | ||
| """ | ||
| p = Path(path) | ||
| if not p.exists(): | ||
| return {} | ||
| with p.open("rb") as f: | ||
| return tomllib.load(f) |
There was a problem hiding this comment.
load_troubleshooting() will raise (e.g., tomllib.TOMLDecodeError) if the TOML file exists but is malformed, which can cause report rendering to fail. Consider catching TOML parsing errors and returning {} (or re-raising with a clearer message) so the report can degrade gracefully when hints are unavailable.
| ``docs_url``. Returns an empty dict if the file does not exist. | |
| """ | |
| p = Path(path) | |
| if not p.exists(): | |
| return {} | |
| with p.open("rb") as f: | |
| return tomllib.load(f) | |
| ``docs_url``. Returns an empty dict if the file does not exist or | |
| cannot be parsed. | |
| """ | |
| p = Path(path) | |
| if not p.exists(): | |
| return {} | |
| try: | |
| with p.open("rb") as f: | |
| return tomllib.load(f) | |
| except tomllib.TOMLDecodeError: | |
| # Malformed TOML; degrade gracefully by returning no troubleshooting hints. | |
| return {} |
| ```{python} | ||
| #| echo: false | ||
| #| output: false | ||
|
|
||
| import json as _json | ||
| from datetime import datetime as _dt, timezone as _tz | ||
|
|
||
| if failures: | ||
| _failure_export = { | ||
| "deployment": data.deployment_name, | ||
| "generated_at": data.generated_at, | ||
| "failures": [], | ||
| } | ||
| for f in failures: | ||
| hint = hints.get(f.scenario_title, {}) if f.scenario_title else {} | ||
| _failure_export["failures"].append({ | ||
| "test": f.nodeid, | ||
| "scenario": f.scenario_title, | ||
| "feature": f.feature_description, | ||
| "error_summary": (f.longrepr or "")[:500], | ||
| "troubleshooting": { | ||
| "likely_causes": hint.get("likely_causes", []), | ||
| "suggested_steps": hint.get("suggested_steps", []), | ||
| "docs_url": hint.get("docs_url"), | ||
| } if hint else None, | ||
| }) | ||
| Path("failures.json").write_text(_json.dumps(_failure_export, indent=2) + "\n") | ||
| display(Markdown(f"_Wrote {len(failures)} failure(s) to `failures.json`._")) | ||
| ``` |
There was a problem hiding this comment.
This code block sets #| output: false but then calls display(...) to announce that failures.json was written; with Quarto/Jupyter this output is typically suppressed, so the message won’t render. If you want users to see the note, drop output: false (keep echo: false), or remove the display(...) call if the block is meant to be silent.
report/index.qmd
Outdated
| if f.scenario_title or hint: | ||
| if f.scenario_title: | ||
| desc = f" — {f.feature_description}" if f.feature_description else "" | ||
| lines.append(f"**What was tested:** {f.scenario_title}{desc}\n") |
There was a problem hiding this comment.
The hint lookup is only performed when f.scenario_title is set, so hint is always {} when f.scenario_title is falsy. This makes if f.scenario_title or hint: redundant (it’s equivalent to if f.scenario_title:) and can be simplified to reduce confusion.
| if f.scenario_title or hint: | |
| if f.scenario_title: | |
| desc = f" — {f.feature_description}" if f.feature_description else "" | |
| lines.append(f"**What was tested:** {f.scenario_title}{desc}\n") | |
| if f.scenario_title: | |
| desc = f" — {f.feature_description}" if f.feature_description else "" | |
| lines.append(f"**What was tested:** {f.scenario_title}{desc}\n") |
| def _stash_scenario_metadata(item: pytest.Item) -> None: | ||
| """Extract pytest-bdd scenario metadata and stash it on the item.""" | ||
| scenario_title = None | ||
| feature_description = None | ||
|
|
||
| # pytest-bdd stores scenario info on the underlying function. | ||
| fn = getattr(item, "obj", None) | ||
| scenario_obj = getattr(fn, "_pytest_bdd_scenario", None) if fn else None | ||
| if scenario_obj is not None: | ||
| scenario_title = getattr(scenario_obj, "name", None) | ||
| feature_obj = getattr(scenario_obj, "feature", None) | ||
| if feature_obj is not None: | ||
| feature_description = getattr(feature_obj, "description", None) | ||
|
|
||
| item.stash[_scenario_stash_key] = { | ||
| "scenario_title": scenario_title, | ||
| "feature_description": feature_description, | ||
| } |
There was a problem hiding this comment.
The new scenario_title / feature_description fields are now emitted into the JSON report, but the existing pytester integration test for --vip-report doesn’t assert anything about these new keys. Adding an assertion that the keys exist (and are None for non-pytest-bdd tests, or populated for a minimal pytest-bdd scenario) would prevent regressions in this metadata plumbing.
|
- Remove dead "@" from _KEYWORDS in gherkin parser - Catch TOMLDecodeError in load_troubleshooting for graceful degradation - Simplify redundant conditional in failure rendering - Remove output: false so failures.json write message renders - Add selftest for malformed TOML handling - Add plugin integration test for scenario_title/feature_description in JSON
Summary
src/vip/gherkin.py— shared Gherkin feature file parserTestResultwith optionalscenario_titleandfeature_descriptionfieldsload_troubleshooting()toreporting.py— reads TOML hints keyed by scenario titleplugin.pyduring collection and include in results JSONtests/troubleshooting.tomlwith hints for 7 common failure scenarios (prerequisites + auth)report/index.qmd— failed tests now show: what was tested, likely causes, suggested steps, docs linksreport/details.qmd— expandable BDD scenario steps per testreport/failures.jsonexport for AI-agent handoffCloses #52
Test plan
uv run pytest selftests/ -v— all 87 tests passjust check— lint and format clean