use a fixture for mss_dir to set a different path for tests

[ReimarBauer]

Purpose of PR?:

Fixes #2958

[matrss]

Are you sure the changes in d67169f are necessary? CI passes without them on macOS runners with the usual (un-)reliability.

[ReimarBauer]

Are you sure the changes in d67169f are necessary? CI passes without them on macOS runners with the usual (un-)reliability.

I recheck, I had first tried a PR with the first none optimal change. Lets see.

[ReimarBauer]

on my macOS 26.2 system I get currently

many "RuntimeError: Server did not start within 5 seconds"

18 passed, 16 skipped, 67 warnings, 358 errors in 84.93s (0:01:24)
I will retry after a fresh reboot.

[ReimarBauer]

=============================================================== short test summary info ===============================================================
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_server_service_cache - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_multilayer_handling - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_filter_handling - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_multilayer_syncing - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_server_getmap - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_invalid_url - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_no_server - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_invalid_schema - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_singlelayer_handling - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52487
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_no_schema - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52487
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_server_getmap_cached - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52487
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_server_no_thread - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52487
ERROR tests/_test_msui/test_sideview.py::Test_SideViewWMS::test_server_getmap - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:52141
========================================= 563 passed, 16 skipped, 92 warnings, 13 errors in 110.51s (0:01:50) =========================================

Here we see an attempt to use 8 times 52141 and 4 times 52487

[ReimarBauer]

Another run 567 passed, 16 skipped, 92 warnings, 9 errors in 113.76s (0:01:53)

ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_invalid_url - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:54919
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_no_server - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:54919
ERROR tests/_test_msui/test_linearview.py::Test_LinearViewWMS::test_server_getmap - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:54977
ERROR tests/_test_msui/test_wms_control.py::Test_VSecWMSControlWidget::test_server_getmap - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:55162
ERROR tests/_test_msui/test_wms_control.py::Test_VSecWMSControlWidget::test_multilayer_drawing - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:55162
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_invalid_schema - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:55162
ERROR tests/_test_msui/test_wms_control.py::Test_HSecWMSControlWidget::test_filter_handling - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:55162
ERROR tests/_test_msui/test_sideview.py::Test_SideViewWMS::test_server_getmap - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:55162
ERROR tests/_test_msui/test_topview.py::Test_TopViewWMS::test_server_getmap - RuntimeError: Server did not start within 5 seconds at http://127.0.0.1:55359

Maybe the problem is that not each test has a different port? here we see 2 times 54919, 5 times 55162

[matrss]

Port selection is not the issue, the OS selects a free port for us which should be race-free even in the presence of many unrelated processes. The reason why multiple errors show the same port number is because those tests share the same fixture instance for the server. E.g. a mswms server fixture is requested at the class-level here:

MSS/tests/_test_msui/test_wms_control.py

Lines 52 to 56 in 27a2fb1

	@pytest.fixture(autouse=True)
	def _with_mswms_server(self, mswms_server):
	self.url = mswms_server
	parsed_url = urllib.parse.urlparse(self.url)
	self.scheme, self.host, self.port = parsed_url.scheme, parsed_url.hostname, parsed_url.port

. This server instance is shared between all tests in that class, or rather its subclasses (but not /between/ subclasses, it seems).

[matrss]

FWIW I know that it passes in CI because I tried it here: https://github.com/matrss/MSS/actions/runs/20426406249/job/58694840047

It just required a bunch of restarts of the macOS 14 tests, which seem to be the most unreliable. But I am not (yet) convinced that the additional changes made it any better.

[ReimarBauer]

FWIW I know that it passes in CI because I tried it here: https://github.com/matrss/MSS/actions/runs/20426406249/job/58694840047

It just required a bunch of restarts of the macOS 14 tests, which seem to be the most unreliable. But I am not (yet) convinced that the additional changes made it any better.

On github there is usually not a ~/mss for the account running the tests. But for a user trying the mswms with demodata has some python config files there.
A user trying mscolab does have there also a colabdata.

Tests should not have the option to change user data or can get different results because of userdata.

[matrss]

I meant the tests pass in CI without the server startup modifications but with the change to the mss_dir configuration, so I don't quite understand your comment...

[ReimarBauer]

I meant the tests pass in CI without the server startup modifications but with the change to the mss_dir configuration, so I don't quite understand your comment...

The tests always pass on GitHub, with or without the change, because we always have a new CI environment there. However, when I run tests locally – which I do frequently – the tests can branch into my local ~/mss directory as observed. In the WayPointsTable and others, we use config_loader(dataset="mss_dir") for the data_dir. If we don't set mss_dir differently, it will use ~/mss.

Let’s treat the problems macOS 26.2 is having with the tests as separate from the related issue. That should be a different issue altogether. It wasn’t a good idea of mine to mix those together here.

[matrss]

The tests always pass on GitHub, with or without the change, because we always have a new CI environment there. However, when I run tests locally – which I do frequently – the tests can branch into my local ~/mss directory as observed. In the WayPointsTable and others, we use config_loader(dataset="mss_dir") for the data_dir. If we don't set mss_dir differently, it will use ~/mss.

Sure, I get that, but how does this relate to what I said? I didn't say we shouldn't set this path to a temporary directory...

Let’s treat the problems macOS 26.2 is having with the tests as separate from the related issue. That should be a different issue altogether. It wasn’t a good idea of mine to mix those together here.

Are there actually any macOS 26.2 specific issues here? AFAICS there is the long-standing issue that macOS tests in general are highly unreliable irrespective of version, but that is unrelated to this PR (and wasn't solved by the server startup modifications that you reverted in the meantime). FWIW it's an approve from me as soon as I see the macOS tests pass just once.

[ReimarBauer]

The tests always pass on GitHub, with or without the change, because we always have a new CI environment there. However, when I run tests locally – which I do frequently – the tests can branch into my local ~/mss directory as observed. In the WayPointsTable and others, we use config_loader(dataset="mss_dir") for the data_dir. If we don't set mss_dir differently, it will use ~/mss.

Sure, I get that, but how does this relate to what I said? I didn't say we shouldn't set this path to a temporary directory...

Let’s treat the problems macOS 26.2 is having with the tests as separate from the related issue. That should be a different issue altogether. It wasn’t a good idea of mine to mix those together here.

Are there actually any macOS 26.2 specific issues here? AFAICS there is the long-standing issue that macOS tests in general are highly unreliable irrespective of version, but that is unrelated to this PR (and wasn't solved by the server startup modifications that you reverted in the meantime). FWIW it's an approve from me as soon as I see the macOS tests pass just once.

My observations are limited to my local environment. After the updates to libgfortran5 and NumPy, I have not observed any differences in daily usage when compared to Ubuntu. Testing no longer hangs. However, it's displaying numerous errors. The amount has changed.

Are you tying this change to the outcome of an unrelated test that's failing intermittently and for unclear reasons?

[ReimarBauer]

FWIW I know that it passes in CI because I tried it here: https://github.com/matrss/MSS/actions/runs/20426406249/job/58694840047

It just required a bunch of restarts of the macOS 14 tests, which seem to be the most unreliable. But I am not (yet) convinced that the additional changes made it any better.

I've realized it wasn't a good idea after all. It would have been nice to fix it with something along those lines, but that was not the solution.

[ReimarBauer]

@matrss Do you see a problem if I define it as (autouse=True)? I'm not sure if it's just another coincidence, but then at least I can run test_open_chat_window without it timing out.

[matrss]

Ahhh, the way it is written now this fixture actually doesn't change anything at all because nothing is requesting it. I don't think making it autouse is a good idea though. Since it only affects msui I think you should add it as another argument to the qtbot fixture (like how it's done for fail_if_open_message_boxes_left and close_remaining_widgets).