Add flag to consider timeouts as "caught"

[sanbox-irl]

Hi there,

For my testing purposes, a timeout of a test is often the equivalent of a "caught" bug. This is true when I run cargo test (if it hangs indefinitely, I do not consider that a success). I respect that not everyone agrees with that, but I'd be nice to have cargo-mutants have an ability to agree.

I believe I could do the equivalent of this by simply checking for exit code 3, but I'd also like the UI to assume that any test caught by the timeout operation is a failed test.

What do you think?

[sourcefrog]

Hi there,

Your analogy that a hung cargo test would effectively be a failure is a good one. Also, because of cargo's current behavior of running tests in parallel and continuing on even if one test fails, it can be difficult to avoid timeouts: even if you have one test that catches the mutation, other tests can be caused to hang.

I was thinking about adding an option like --accept=timeout which causes it to return 0 even if some tests time out. This could extend to also --accept=timeouts,missed for cases like CI where you really care more about distinguishing complete collection of results from errors.

I'd also like the UI to assume that any test caught by the timeout operation is a failed test.

Can you say more about this? Do you want it to print "missed" instead of "timeout", or something else?

[sanbox-irl]

I was thinking about adding an option like --accept=timeout which causes it to return 0 even if some tests time out. This could extend to also --accept=timeouts,missed for cases like CI where you really care more about distinguishing complete collection of results from errors.

This is exactly what I'd want! Sorry when I said "UI" (maybe from staying up a bit too late...), I meant the CLI's response. If we have an --accept=timeout, I'd want the UI to conclude a timeout is a "caught" mutant.

[sanbox-irl]

If you can direct me towards that, I'd also be willing to do that work (including updating the docs, etc!)

[sourcefrog]

Sure, so basically you need to:

• (Think if there's a better name than "accept")
• Add accept: Vec<SummaryOutcome> into Options in options.rs
• Add it to Args in main.rs, and then to the code that creates Options from Args
• Similarly add it to Config so it can be in the config file and feed that in to Options; add unit tests for this like the existing ones
• Maybe derive_more::FromStr for SummaryOutcome
• Add to book/ somewhere reasonable; check it with mdbook open
• Use it in LabOutcome::exit_code; test this; might need to pass in the options
• Add a CLI integration test but be careful that the test it runs does not make it too slow

[sourcefrog]

ps: nice cat avi!