-
Notifications
You must be signed in to change notification settings - Fork 34
Description
phd-tests/runner/src/main.rs is where we produce the nice test result: ... X passed; Y failed; Z skipped; W not run; finished in ... line, but actual numbers of pass/fail/skipped vary by guest fixture.
while one would hope there are no failures, some tests involve Linux-isms that don't survive contact with cmd.exe and fail for boring reasons, while other tests skip on guest OSes that don't provide all the desired test facilities (usually some tool not present on Alpine like lshw). finally, at least the CPU topology test is (currently) Intel-only and Linux-only.
it'd be nice for phd-runner to, when running all tests, assert that the number of pass/fail/skip actually match our expectations for the given test image adapter, hardware, etc. if a test moves from passing to skipped without us knowing, that's bad!