Supporting packages whose tests depend on Preferences

[timholy]

In principle it is possible that some packages may rely on configuration to pass their tests. If that configuration occurs via Preferences.jl, given that Preferences is path-based it seems likely to be necessary to find and copy the LocalPreferences.toml file to the new project directory.

I don't yet have a MWE, but I have a not-yet-public package that involves communication with a SQL database, and I handle the configuration of this interaction via Preferences. The package passes tests with Pkg.test() (following a custom line in my CI.yml file that generates the right LocalPreferences) but I have not found an incantation that works with TestEnv.

[oxinabox]

We will have to locate which thing is present in Pkg.test that is missing from TestEnv.

[timholy]

I think for preferences that's a question of traversing the same preferences path that the loading system uses. I'm not deeply (or even shallowly 🙂 ) familiar with that part of base/loading.jl but it's presumably not too complicated.

[jkrumbiegel]

I wanted to voice support for this issue as I usually disable precompilation for packages I develop, via preferences. Those are then not picked up anymore when using TestEnv. It's not critical for functioning but it slows down the process.

[oxinabox]

I tried to workout where in the Pkg.test code this was being enabled but couldn't find it. If someone can point out the Pkg code that ties this I can copy it across

[moble]

I don't understand how any of this works, but here's one simple example. It's related to ForwardDiff, which I use just as a quick-and-dirty test for some analytic derivatives. But I need the "NaN-safe mode", which apparently can't be enabled dynamically, but must be already turned on in LocalPreferences.toml when you load ForwardDiff:

[ForwardDiff]
nansafe_mode = true

I have this in my test directory, but once I run TestEnv.activate(), it's not found, so my tests fail. The MWE for that is just

log(ForwardDiff.Dual{:tag}(0.0, 0.0))

The "wrong" answer (returned when nansafe_mode is not enabled) is Dual{:tag}(-Inf,NaN); the "right" answer is Dual{:tag}(-Inf,0.0). My hacky workaround is to copy LocalPreferences.toml from test to the directory returned by activate:

cp("test/LocalPreferences.toml", joinpath(dirname(TestEnv.activate()),"LocalPreferences.toml"))

Love this package, BTW! :)

[oxinabox]

This might actually be the correct solution (if we did that automatically).
I don't understand how the preferences stuff works at all.

[jkrumbiegel]

As far as I understand, there can either be keys stored in a sidecar file, (Julia)LocalPreferences.toml, or in the project file Project.toml. I assume the keys in the project file are already copied over currently, but I haven't checked. So only the other file needs to be copied over. What complicates things a bit is that preferences can be inherited from envs higher up the load path:

Preferences can be set with depot-wide defaults; if package Foo is installed within your global environment and it has preferences set, these preferences will apply as long as your global environment is part of your LOAD_PATH. Preferences in environments higher up in the environment stack get overridden by the more proximal entries in the load path, ending with the currently active project. This allows depot-wide preference defaults to exist, with active projects able to merge or even completely overwrite these inherited preferences.

So you would only get the exact same behavior if at the time you run tests manually after having TestEnv.activate()d, your LOAD_PATH is the same as it would be when running ]test.

[jkrumbiegel]

From a quick test with a random package that prints out the load path during testing, I get this in the normal REPL session:

julia> LOAD_PATH
3-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"

And this from the ]test invocation:

LOAD_PATH = ["@", "/var/folders/z5/r5q6djwn5g10k3w279bn37700000gn/T/jl_U1R2Hh"]

As the paths differ, preferences might be different between ]test and TestEnv.activate() even if LocalPreferences.toml is copied over. So TestEnv.activate() might need to modify LOAD_PATH? Currently it does not.

On the other hand, now that I think about it, you would not want your tests to be vulnerable to some random dependency's preferences, so if there is a setting that must not be changed from within a higher-level env, then you should probably set that preference yourself in LocalPreferences.toml. In this case, it would be enough to copy that one over and say "any other conflict with preferences is your responsibility".

[devmotion]

I ran into this problem as well. Mandatory preferences for a Julia package that I keep in a LocalPreferences.toml file in the package repo (only locally, not checked in into git) are picked up and used when running Pkg.test but not available when working with TestEnv. As a (quite inconvenient) workaround, currently I copy the preference file manually to the path of the environment created by TestEnv.

When creating the sandbox test environment, Pkg collects all preferences
https://github.com/JuliaLang/Pkg.jl/blob/f7f222f28e929dda69e4f5b1d660fc51a133fe09/src/Operations.jl#L1953-L1964
and writes them to a JuliaLocalPreferences.toml file
https://github.com/JuliaLang/Pkg.jl/blob/f7f222f28e929dda69e4f5b1d660fc51a133fe09/src/Operations.jl#L1757
https://github.com/JuliaLang/Pkg.jl/blob/f7f222f28e929dda69e4f5b1d660fc51a133fe09/src/Operations.jl#L1798-L1803
I think ideally TestEnv would follow the same approach and write all available preferences to a (Julia)LocalPreferences.toml file in its temporary environment.

[oxinabox]

I agree that would be the correct approach.
Thank you for finding where the code we need to steal is.
A PR would be accepted to do that.
I am pretty busy right now, and unlikely to get to it myself for quite a while