Support filtering or selecting tests by filename

[reify-tanner-stirrat]

The use case

CircleCI has functionality for getting a list of test names and then distributing them across parallel test runners. It's relatively easy to make this work with test runners in other languages, because many of them provide functionality for providing a list of test files to be interpreted and run by the runner. I'd like to be able to use this somehow.

Additionally, focusing by file would make it relatively easy to quickly run a subset of tests just based on file structure from the command line without modifying any files, which to my knowledge isn't possible.

Existing workarounds

Writing a kaocha entrypoint that bashes out to the CircleCI binary

You can see an example of it here: https://andreacrotti.github.io/2020-07-28-parallel-ci-kaocha/

This feels brittle compared to providing an arglist - it's structuring a runner around shelling out rather than providing an external interface that can be piped or arg'd in.

Writing a plugin

I've currently gone this route. I was able to write a plugin that takes --focus-files and then hacks the options of kaocha.plugin/filter to provide ns filtering indirectly. This works but is brittle because it depends on the order of plugin application.

I could also go and reimplement enough of the filter plugin's functionality to make the plugin self-contained, but that seems like a duplication of effort.

Proposed solutions

Add --focus-files as an option to the filter plugin

This could live in the filter plugin and use some of plugin's existing functionality. It seems to fit with the concepts of the other filter mechanisms.

Make fully-qualified files valid entries for :test-paths

If fully-qualified files were an option here, it would be possible to write a plugin that takes a --focus-files (or similar) argument and updates the :test-paths vector with those files. This seems like it would have an additional benefit of cutting down on the number of namespaces that are actually loaded by the framework before tests begin.

Conclusion

Let me know if any of these approaches is better than another in your eyes. I'm happy to provide a PR.

[plexus]

Thanks for the suggestion, @reify-tanner-stirrat , this seems like a good feature to have. Are you merely suggesting that someone do this, or are you volunteering to shepherd this feature? Either way I'll add it to our project board for the upcoming Clojurists Together funding.

I think we can do a relatively simple thing here where we add a top-level hint in the config about which files to load, based on the --focus-filter flag, which subsequently test suites can use to limit what they load.

:test-paths is not suitable here because we don't know which :test-paths to add them to, that's why I think it should be a top-level key.

Making sure this happens in the load stage also makes sense because it's the only place where we deal with things at the file level, once things are loaded we inspect the clojure runtime for namespaces and vars. We can map these back to resources on the classpath, and from there map them back to files, but that's going the long way round.

Note that while this would be fairly easy for clojure.test, a full solution should also address other test types: cljs, cljs2, cucumber, and potentially midje (we are not committed to maintaining midje, but in this case the change should be similar to what's needed for clojure.test.)

[reify-tanner-stirrat]

I was planning on starting work on this if it was something y'all are keen to see.

I think we can do a relatively simple thing here where we add a top-level hint in the config about which files to load, based on the --focus-filter flag, which subsequently test suites can use to limit what they load.

So the recommended approach is something like:

• Turn --focus-files into a :kaocha/test-files key on the config object
• Modify the load logic so that certain testables are not generated or discarded?

One thing I'm unclear on is what is actually loading/interpreting the files, and I was expecting to see something that is recursively walking a directory somewhere, which I couldn't find. Is there a different mechanism at play?

Additionally, my current understanding of the clojure tools cli API is that you can't provide whitespace-separated arguments to a single flag, a la:

kaocha --focus-files foo/bar foo/baz foo/qux

which would really be the ideal case for piping in a whitespace-separated list of files from another source. What's the most common approach for that? Comma separation?

[sebastian-lerner]

@plexus any update on this?