Merged
Conversation
Backstop for the no-skips-in-CI policy. The conftest hook rewrites any test reported as skipped into a setup error when GITHUB_ACTIONS=true, which propagates to a non-zero pytest exit code through normal failure accounting. Local dev is unchanged — the env var is GH-Actions-specific. Now if a future test gains a skip marker without its tool being added to CI alongside (the trap that #54 just plugged for cue), the very next CI run fails loudly instead of silently passing with the test having never executed. Implementation: - pytest_runtest_makereport hookwrapper in tests/conftest.py - Watches for `report.skipped and report.when == "setup"` (where skipif markers fire) under GITHUB_ACTIONS=true - Rewrites outcome to "failed" with a CI-policy explanation, leaving the original skip reason inline for traceability - Verified locally: env unset → 380 passed; env set with no skips → 380 passed; env set with a synthetic skipped test → exit 1 with the policy violation rendered as a setup error Closes #55.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backstop for the no-skips-in-CI policy established in #53/#54. The pytest hook in `tests/conftest.py` rewrites any `skipped` test report into a setup `error` when running under `GITHUB_ACTIONS=true`. Local development is unchanged — the env var only fires inside our actual CI.
Why
After #54 installed cue in CI to remove the `requires_cue` skips, the underlying gap was still there: any future test that gains a new skip marker without its corresponding CI tool would silently not run, and CI would still go green. This PR closes that gap structurally — the next time a skip would happen in CI, the run fails with an explicit "CI policy: skipped tests are forbidden" message and the original skip reason inline.
Verification
Implementation
```python
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
if os.environ.get("GITHUB_ACTIONS") != "true":
return outcome
report = outcome.get_result()
if report.skipped and report.when == "setup":
report.outcome = "failed"
report.longrepr = (
f"CI policy: skipped tests are forbidden under GITHUB_ACTIONS=true. "
f"Install the required tool in CI instead of skipping. "
f"Original skip reason: {original}"
)
```
`when == "setup"` is where skipif markers fire — the standard `pytest.skip()` calls inside test bodies report at the `call` phase, so this hook intentionally only catches the markers. If we want to cover both later, that's a one-line change.
Scope
Why Not Earlier Approaches
The first attempt set `session.exitstatus = 1` in `pytest_sessionfinish` — doesn't take effect in pytest 9. The second tried `pytest.exit(returncode=1)` — also got swallowed. Rewriting the report itself in `pytest_runtest_makereport` is the cleanest path: pytest's own failure accounting handles the rest.
Closes #55.