Skip to content

feat(ci): hard-fail CI on test skips#56

Merged
f3rdy merged 1 commit intomasterfrom
feature/55-fail-ci-on-skip
Apr 28, 2026
Merged

feat(ci): hard-fail CI on test skips#56
f3rdy merged 1 commit intomasterfrom
feature/55-fail-ci-on-skip

Conversation

@f3rdy
Copy link
Copy Markdown
Contributor

@f3rdy f3rdy commented Apr 28, 2026

Summary

Backstop for the no-skips-in-CI policy established in #53/#54. The pytest hook in `tests/conftest.py` rewrites any `skipped` test report into a setup `error` when running under `GITHUB_ACTIONS=true`. Local development is unchanged — the env var only fires inside our actual CI.

Why

After #54 installed cue in CI to remove the `requires_cue` skips, the underlying gap was still there: any future test that gains a new skip marker without its corresponding CI tool would silently not run, and CI would still go green. This PR closes that gap structurally — the next time a skip would happen in CI, the run fails with an explicit "CI policy: skipped tests are forbidden" message and the original skip reason inline.

Verification

  • `uv run pytest` (no env): 380 passed, 88% coverage. Local dev unchanged.
  • `GITHUB_ACTIONS=true uv run pytest` against current suite: 380 passed (no skips fire because cue + ansible-vault are installed).
  • `GITHUB_ACTIONS=true uv run pytest tests/<synthetic_skip>.py`: exit 1, setup error with the policy message and original skip reason. Verified before commit.

Implementation

```python
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
if os.environ.get("GITHUB_ACTIONS") != "true":
return outcome
report = outcome.get_result()
if report.skipped and report.when == "setup":
report.outcome = "failed"
report.longrepr = (
f"CI policy: skipped tests are forbidden under GITHUB_ACTIONS=true. "
f"Install the required tool in CI instead of skipping. "
f"Original skip reason: {original}"
)
```

`when == "setup"` is where skipif markers fire — the standard `pytest.skip()` calls inside test bodies report at the `call` phase, so this hook intentionally only catches the markers. If we want to cover both later, that's a one-line change.

Scope

  • One hook in `tests/conftest.py` (~25 lines including doc).
  • One CLAUDE.md note under Testing Strategy.
  • No exemptions, no allow-list. New skip markers fail until paired with their CI tool — exactly the contract.

Why Not Earlier Approaches

The first attempt set `session.exitstatus = 1` in `pytest_sessionfinish` — doesn't take effect in pytest 9. The second tried `pytest.exit(returncode=1)` — also got swallowed. Rewriting the report itself in `pytest_runtest_makereport` is the cleanest path: pytest's own failure accounting handles the rest.

Closes #55.

Backstop for the no-skips-in-CI policy. The conftest hook rewrites any
test reported as skipped into a setup error when GITHUB_ACTIONS=true,
which propagates to a non-zero pytest exit code through normal failure
accounting. Local dev is unchanged — the env var is GH-Actions-specific.

Now if a future test gains a skip marker without its tool being added
to CI alongside (the trap that #54 just plugged for cue), the very next
CI run fails loudly instead of silently passing with the test having
never executed.

Implementation:
- pytest_runtest_makereport hookwrapper in tests/conftest.py
- Watches for `report.skipped and report.when == "setup"` (where skipif
  markers fire) under GITHUB_ACTIONS=true
- Rewrites outcome to "failed" with a CI-policy explanation, leaving
  the original skip reason inline for traceability
- Verified locally: env unset → 380 passed; env set with no skips →
  380 passed; env set with a synthetic skipped test → exit 1 with the
  policy violation rendered as a setup error

Closes #55.
@f3rdy f3rdy merged commit 1e4488e into master Apr 28, 2026
2 checks passed
@f3rdy f3rdy deleted the feature/55-fail-ci-on-skip branch April 28, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hard-fail CI on test skips (no skips allowed)

1 participant