Init time takes longer and longer on macOS

[giordano]

Now that #86 shows init time, I noticed that macOS jobs seem to take longer and longer to do the init, at least in Breeze.jl. This happens only on macOS, and only in julia v1.12, no problem on other platforms or previous versions.

See for example https://github.com/NumericalEarth/Breeze.jl/actions/runs/23542325841/job/68532917031, with ParallelTestRunner v2.5.0:

Running 31 tests using 3 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
Available memory: 4.285 GiB
                                               │   Test   │   Init   │ Compile │ ──────────────── CPU ──────────────── │
Test                                  (Worker) │ time (s) │ time (s) │   (%)   │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
quality_assurance                          (1) │        started at 2026-03-25T13:08:30.688
doctests                                   (2) │        started at 2026-03-25T13:08:30.697
forcing_and_boundary_conditions            (3) │        started at 2026-03-25T13:08:30.708
quality_assurance                          (1) │   115.53 │     5.38 │   28.89 │  12.84 │ 11.1 │    7485.34 │  1449.09 │
cloud_microphysics_1M                      (1) │        started at 2026-03-25T13:10:32.738
doctests                                   (2) │   160.49 │     5.40 │   94.64 │  20.58 │ 12.8 │   19353.32 │  1562.33 │
advection_schemes                          (2) │        started at 2026-03-25T13:11:20.028
forcing_and_boundary_conditions            (3) │   173.82 │     5.39 │   98.85 │  22.35 │ 12.9 │   20766.84 │  1200.52 │
atmosphere_model_construction              (3) │        started at 2026-03-25T13:11:29.920
advection_schemes                          (2) │    89.98 │     1.27 │   98.18 │  30.90 │ 34.3 │    6358.87 │  1562.33 │
acoustic_substepping                       (2) │        started at 2026-03-25T13:12:53.616
atmosphere_model_construction              (3) │    99.88 │     0.91 │   98.69 │  25.63 │ 25.7 │    6265.11 │  1321.16 │
radiation_scheduling                       (3) │        started at 2026-03-25T13:13:14.535
radiation_scheduling                       (3) │    57.12 │     1.11 │   98.73 │  17.15 │ 30.0 │    2525.93 │  1347.94 │
diagnostics                                (3) │        started at 2026-03-25T13:14:25.257
cloud_microphysics_1M                      (1) │   234.48 │     2.57 │   90.83 │  75.35 │ 32.1 │   15601.75 │  1602.69 │
cloud_microphysics_2M                      (1) │        started at 2026-03-25T13:14:37.222
diagnostics                                (3) │    39.68 │     2.46 │   97.02 │   8.96 │ 22.6 │    2037.67 │  1513.05 │
gray_radiative_transfer                    (3) │        started at 2026-03-25T13:15:14.921
gray_radiative_transfer                    (3) │     4.99 │    10.14 │   93.23 │   0.00 │  0.0 │     554.82 │  1513.05 │
dynamics                                   (3) │        started at 2026-03-25T13:16:00.910
cloud_microphysics_2M                      (1) │   150.46 │    22.00 │   95.06 │  38.40 │ 25.5 │    6441.16 │  1602.69 │
vertical_diffusion                         (1) │        started at 2026-03-25T13:17:59.225
vertical_diffusion                         (1) │    41.25 │    29.75 │   82.15 │   9.59 │ 23.2 │    2785.63 │  1602.69 │
turbulence_closures                        (1) │        started at 2026-03-25T13:19:15.474
dynamics                                   (3) │   313.34 │    21.23 │   97.07 │ 104.36 │ 33.3 │   13470.19 │  1513.05 │
set_atmosphere_model                       (3) │        started at 2026-03-25T13:22:31.799
acoustic_substepping                       (2) │   513.63 │     0.88 │   91.36 │ 216.56 │ 42.2 │   18608.06 │  1562.33 │
reference_states                           (2) │        started at 2026-03-25T13:22:35.860
set_atmosphere_model                       (3) │    72.30 │    67.18 │   97.18 │  12.88 │ 17.8 │    2639.19 │  1513.05 │
piecewise_linear_stretching                (3) │        started at 2026-03-25T13:26:07.826
reference_states                           (2) │    58.31 │    75.63 │   96.35 │   7.66 │ 13.1 │    2594.06 │  1562.33 │
saturation_adjustment                      (2) │        started at 2026-03-25T13:26:14.343
turbulence_closures                        (1) │   388.46 │    47.98 │   97.76 │ 125.54 │ 32.3 │   13686.57 │  1602.69 │
geostrophic_subsidence_forcings            (1) │        started at 2026-03-25T13:27:54.536
piecewise_linear_stretching                (3) │     7.34 │    90.48 │   89.00 │   0.00 │  0.0 │     168.40 │  1513.05 │
anelastic_pressure_solver_nonhydrostatic   (3) │        started at 2026-03-25T13:29:00.351
saturation_adjustment                      (2) │    45.24 │    99.47 │   94.94 │   0.00 │  0.0 │    1238.75 │  1562.33 │
all_sky_radiative_transfer                 (2) │        started at 2026-03-25T13:30:15.712
geostrophic_subsidence_forcings            (1) │   278.41 │    62.09 │   97.08 │  51.79 │ 18.6 │    4559.11 │  1602.69 │
parcel_dynamics                            (1) │        started at 2026-03-25T13:37:25.879
anelastic_pressure_solver_nonhydrostatic   (3) │   195.17 │    83.38 │   98.72 │  36.96 │ 18.9 │    2898.02 │  1513.05 │
moist_air_buoyancy                         (3) │        started at 2026-03-25T13:37:56.860
all_sky_radiative_transfer                 (2) │   249.60 │   101.08 │   82.38 │  74.94 │ 30.0 │    3664.86 │  1562.33 │
clear_sky_radiative_transfer               (2) │        started at 2026-03-25T13:40:20.490
moist_air_buoyancy                         (3) │   121.56 │   220.69 │   96.34 │   0.00 │  0.0 │    1354.68 │  1513.05 │
anelastic_pressure_solver_analytic         (3) │        started at 2026-03-25T13:48:58.767
clear_sky_radiative_transfer               (2) │    51.07 │   186.16 │   71.17 │   0.00 │  0.0 │     726.42 │  1562.33 │
tracer_dynamics                            (2) │        started at 2026-03-25T13:49:26.771
parcel_dynamics                            (1) │   269.34 │   203.43 │   87.55 │  62.08 │ 23.0 │    1785.16 │  1602.69 │
dcmip2016_kessler                          (1) │        started at 2026-03-25T13:49:49.006
tracer_dynamics                            (2) │    89.66 │   390.06 │   97.32 │   0.00 │  0.0 │    1124.55 │  1562.33 │
cloud_microphysics_0M                      (2) │        started at 2026-03-25T14:05:08.801
dcmip2016_kessler                          (1) │   159.18 │   350.07 │   97.14 │   0.00 │  0.0 │    1282.84 │  1602.69 │
kinematic_driver                           (1) │        started at 2026-03-25T14:05:19.154
anelastic_pressure_solver_analytic         (3) │   187.24 │   403.36 │   97.93 │   0.00 │  0.0 │    1321.24 │  1513.05 │
unit_tests                                 (3) │        started at 2026-03-25T14:06:05.407
unit_tests                                 (3) │    14.03 │   309.43 │   68.06 │   0.00 │  0.0 │      90.72 │  1513.05 │
polynomial_bulk_coefficients               (3) │        started at 2026-03-25T14:15:43.031
Error: The operation was canceled.

[christiangnrd]

Are you forcing 3 parallel jobs?

[giordano]

Are you forcing 3 parallel jobs?

Yes, do you think that's the problem?

[christiangnrd]

Yeah. I had to remove it in GPUArrays because of this. #86 was written because I was trying to figure out why the reported testset times looked similar to 1.10 but felt so much slower.

[giordano]

Do you think it's a memory issue, slowing everything to a crawl? In this case apparently at the beginning there are 4.2 GB available, in other runs it's a little bit more than 5. I increased the number of jobs from 2 to 3 in NumericalEarth/Breeze.jl#514, because overall build time with 2 jobs was a bit too much, but moving to 3 didn't seem to improve things much (and we didn't have the extra information printed by #86 back then)

[christiangnrd]

While I didn't explicitly confirm, I think when the pre-test times baloon it's because of memory thrashing

[giordano]

This job with Julia v1.11 finished in 17 minutes, instead of timing out after 80 minutes, and funnily enough at the beginning had

Running 32 tests using 3 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
Available memory: 3.277 GiB

which is even less available memory than above. This also showed longer init times for the final tests, but in the order of 30 seconds, not 300-400.

Alright, I guess I'll reduce number of jobs back to 2, and hope things improve.

[giordano]

Reducing number of jobs to 2, helped a little bit, but the increasing init times are still there:
julia v1.11:

Running 32 tests using 2 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
Available memory: 5.072 GiB
                                               │   Test   │   Init   │ Compile │ ──────────────── CPU ──────────────── │
Test                                  (Worker) │ time (s) │ time (s) │   (%)   │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
piecewise_linear_stretching                (1) │        started at 2026-04-01T17:09:11.665
cloud_microphysics_2M                      (2) │        started at 2026-04-01T17:09:11.668
piecewise_linear_stretching                (1) │     2.00 │     6.04 │   93.86 │   0.08 │  4.0 │     389.88 │   730.11 │
reference_states                           (1) │        started at 2026-04-01T17:09:19.113
reference_states                           (1) │    34.75 │     0.36 │   98.26 │   2.31 │  6.7 │    4753.14 │  1114.03 │
all_sky_radiative_transfer                 (1) │        started at 2026-04-01T17:09:54.655
all_sky_radiative_transfer                 (1) │    25.35 │     0.31 │   75.14 │   1.59 │  6.3 │    4231.25 │  1617.41 │
polynomial_bulk_coefficients               (1) │        started at 2026-04-01T17:10:20.591
polynomial_bulk_coefficients               (1) │     0.50 │     0.35 │   67.84 │   0.00 │  0.0 │      45.00 │  1617.41 │
moist_air_buoyancy                         (1) │        started at 2026-04-01T17:10:21.622
moist_air_buoyancy                         (1) │     7.29 │     0.26 │   99.44 │   0.36 │  4.9 │    2688.58 │  1639.30 │
terrain_following                          (1) │        started at 2026-04-01T17:10:29.482
cloud_microphysics_2M                      (2) │    76.25 │     6.29 │   98.20 │   4.10 │  5.4 │   11960.75 │  1253.31 │
dcmip2016_kessler                          (2) │        started at 2026-04-01T17:10:33.636
dcmip2016_kessler                          (2) │     7.85 │     0.29 │   99.07 │   0.18 │  2.3 │    1346.95 │  1437.17 │
parcel_dynamics                            (2) │        started at 2026-04-01T17:10:42.071
terrain_following                          (1) │    13.12 │     0.25 │   98.13 │   0.36 │  2.8 │    2383.00 │  1639.30 │
dynamics                                   (1) │        started at 2026-04-01T17:10:43.418
parcel_dynamics                            (2) │     9.84 │     0.27 │   83.58 │   0.25 │  2.5 │    2027.55 │  1448.80 │
diagnostics                                (2) │        started at 2026-04-01T17:10:52.739
diagnostics                                (2) │    14.23 │     0.32 │   97.91 │   0.57 │  4.0 │    2063.21 │  1503.58 │
kinematic_driver                           (2) │        started at 2026-04-01T17:11:07.789
kinematic_driver                           (2) │    10.62 │     0.39 │   87.06 │   0.40 │  3.7 │    1855.13 │  1594.72 │
turbulence_closures                        (2) │        started at 2026-04-01T17:11:19.303
dynamics                                   (1) │    59.28 │     0.40 │   92.33 │   4.88 │  8.2 │   12841.10 │  1807.03 │
anelastic_pressure_solver_analytic         (1) │        started at 2026-04-01T17:11:43.723
anelastic_pressure_solver_analytic         (1) │     6.55 │     0.42 │   99.14 │   0.31 │  4.7 │    1059.14 │  1807.03 │
tracer_dynamics                            (1) │        started at 2026-04-01T17:11:51.305
tracer_dynamics                            (1) │     6.63 │     0.54 │   98.52 │   0.30 │  4.6 │    1152.64 │  1807.03 │
quality_assurance                          (1) │        started at 2026-04-01T17:11:59.365
turbulence_closures                        (2) │    96.14 │     0.34 │   98.59 │   6.94 │  7.2 │   15443.03 │  1839.11 │
geostrophic_subsidence_forcings            (2) │        started at 2026-04-01T17:12:56.668
geostrophic_subsidence_forcings            (2) │    29.72 │     0.46 │   98.30 │   1.56 │  5.2 │    3589.27 │  1839.11 │
vertical_diffusion                         (2) │        started at 2026-04-01T17:13:28.173
quality_assurance                          (1) │    93.46 │     0.49 │   43.53 │   9.56 │ 10.2 │    8364.80 │  2048.81 │
gray_radiative_transfer                    (1) │        started at 2026-04-01T17:13:35.376
gray_radiative_transfer                    (1) │     8.21 │     1.52 │   97.12 │   0.69 │  8.4 │    1080.03 │  2068.45 │
saturation_adjustment                      (1) │        started at 2026-04-01T17:13:46.850
vertical_diffusion                         (2) │    24.09 │     0.56 │   91.99 │   2.68 │ 11.1 │    2587.97 │  1839.11 │
cloud_microphysics_1M                      (2) │        started at 2026-04-01T17:13:53.726
saturation_adjustment                      (1) │    17.29 │     0.74 │   98.25 │   1.34 │  7.8 │    1315.16 │  2068.45 │
set_atmosphere_model                       (1) │        started at 2026-04-01T17:14:06.889
set_atmosphere_model                       (1) │    43.72 │     0.84 │   98.42 │   5.64 │ 12.9 │    2553.73 │  2098.11 │
acoustic_substepping                       (1) │        started at 2026-04-01T17:14:56.724
cloud_microphysics_1M                      (2) │    88.40 │     0.46 │   93.12 │  11.91 │ 13.5 │    6894.26 │  1839.11 │
anelastic_pressure_solver_nonhydrostatic   (2) │        started at 2026-04-01T17:15:24.371
anelastic_pressure_solver_nonhydrostatic   (2) │    26.59 │     0.99 │   97.82 │   3.28 │ 12.3 │    3079.28 │  1839.11 │
clear_sky_radiative_transfer               (2) │        started at 2026-04-01T17:15:55.646
clear_sky_radiative_transfer               (2) │    34.36 │     1.24 │   84.74 │   4.38 │ 12.8 │    2393.95 │  1839.11 │
unit_tests                                 (2) │        started at 2026-04-01T17:16:33.288
unit_tests                                 (2) │     3.25 │     0.62 │   90.75 │   0.00 │  0.0 │      98.41 │  1839.11 │
cloud_microphysics_0M                      (2) │        started at 2026-04-01T17:16:38.950
cloud_microphysics_0M                      (2) │    23.95 │     0.65 │   98.28 │   2.47 │ 10.3 │    1054.19 │  1839.11 │
radiation_scheduling                       (2) │        started at 2026-04-01T17:17:05.956
radiation_scheduling                       (2) │    13.86 │     1.87 │   98.40 │   1.24 │  8.9 │    1103.70 │  1839.11 │
forcing_and_boundary_conditions            (2) │        started at 2026-04-01T17:17:23.854
acoustic_substepping                       (1) │   282.18 │     2.13 │   85.91 │  71.28 │ 25.3 │   16510.96 │  2098.11 │
atmosphere_model_construction              (1) │        started at 2026-04-01T17:19:58.589
atmosphere_model_construction              (1) │   116.51 │    12.83 │   98.14 │  16.89 │ 14.5 │    4641.47 │  2098.11 │
doctests                                   (1) │        started at 2026-04-01T17:22:31.033
forcing_and_boundary_conditions            (2) │   374.49 │     0.94 │   97.68 │  51.00 │ 13.6 │   14197.11 │  1892.39 │
advection_schemes                          (2) │        started at 2026-04-01T17:23:53.752
advection_schemes                          (2) │   156.85 │     3.75 │   98.61 │  36.90 │ 23.5 │    6582.11 │  1892.39 │
doctests                                   (1) │   242.25 │    23.01 │   90.45 │  53.78 │ 22.2 │    9444.75 │  2783.38 │

julia v1.12:

Running 32 tests using 2 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
Available memory: 4.512 GiB
                                               │   Test   │   Init   │ Compile │ ──────────────── CPU ──────────────── │
Test                                  (Worker) │ time (s) │ time (s) │   (%)   │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
geostrophic_subsidence_forcings            (1) │        started at 2026-04-01T17:13:50.838
radiation_scheduling                       (2) │        started at 2026-04-01T17:13:50.842
radiation_scheduling                       (2) │    25.47 │     6.58 │   98.58 │   1.97 │  7.7 │    4411.03 │  1114.88 │
all_sky_radiative_transfer                 (2) │        started at 2026-04-01T17:14:22.332
geostrophic_subsidence_forcings            (1) │    46.07 │     6.62 │   99.18 │   2.76 │  6.0 │    8060.21 │  1198.42 │
clear_sky_radiative_transfer               (1) │        started at 2026-04-01T17:14:42.894
all_sky_radiative_transfer                 (2) │    25.20 │     0.51 │   76.31 │   2.21 │  8.8 │    4618.43 │  1654.55 │
dcmip2016_kessler                          (2) │        started at 2026-04-01T17:14:48.525
dcmip2016_kessler                          (2) │     8.79 │     0.52 │   98.84 │   0.34 │  3.9 │    1580.03 │  1673.61 │
kinematic_driver                           (2) │        started at 2026-04-01T17:14:58.266
clear_sky_radiative_transfer               (1) │    19.10 │     0.69 │   85.19 │   1.09 │  5.7 │    3332.20 │  1567.69 │
unit_tests                                 (1) │        started at 2026-04-01T17:15:03.252
unit_tests                                 (1) │     1.70 │     0.57 │   84.48 │   0.00 │  0.0 │     221.99 │  1567.69 │
doctests                                   (1) │        started at 2026-04-01T17:15:06.032
kinematic_driver                           (2) │    65.00 │     0.56 │   97.65 │   6.23 │  9.6 │    8502.97 │  1866.03 │
quality_assurance                          (2) │        started at 2026-04-01T17:16:05.486
doctests                                   (1) │   157.49 │     0.50 │   68.18 │  10.37 │  6.6 │   14672.41 │  1651.28 │
anelastic_pressure_solver_nonhydrostatic   (1) │        started at 2026-04-01T17:17:45.205
anelastic_pressure_solver_nonhydrostatic   (1) │    17.35 │     0.93 │   96.48 │   1.29 │  7.4 │    2123.76 │  1845.41 │
acoustic_substepping                       (1) │        started at 2026-04-01T17:18:05.215
quality_assurance                          (2) │   206.84 │     0.61 │   17.78 │  13.34 │  6.4 │    8319.51 │  1903.64 │
cloud_microphysics_1M                      (2) │        started at 2026-04-01T17:19:34.893
acoustic_substepping                       (1) │   174.23 │     0.86 │   84.51 │  28.98 │ 16.6 │   18486.77 │  1980.70 │
atmosphere_model_construction              (1) │        started at 2026-04-01T17:21:06.235
cloud_microphysics_1M                      (2) │   138.76 │     2.65 │   90.24 │  37.24 │ 26.8 │   16299.04 │  2153.80 │
turbulence_closures                        (2) │        started at 2026-04-01T17:21:58.421
atmosphere_model_construction              (1) │    52.09 │     1.43 │   98.35 │  11.60 │ 22.3 │    4480.94 │  1980.70 │
set_atmosphere_model                       (1) │        started at 2026-04-01T17:22:02.959
set_atmosphere_model                       (1) │    17.04 │     1.01 │   97.98 │   2.79 │ 16.4 │    1851.94 │  1980.70 │
vertical_diffusion                         (1) │        started at 2026-04-01T17:22:29.947
vertical_diffusion                         (1) │    23.85 │     0.84 │   82.84 │   6.50 │ 27.3 │    2645.40 │  1980.70 │
terrain_following                          (1) │        started at 2026-04-01T17:22:56.390
terrain_following                          (1) │    25.79 │     6.14 │   98.07 │   5.63 │ 21.8 │    2331.37 │  1980.70 │
saturation_adjustment                      (1) │        started at 2026-04-01T17:23:36.356
saturation_adjustment                      (1) │    10.22 │     1.87 │   95.52 │   0.00 │  0.0 │    1120.70 │  2029.50 │
parcel_dynamics                            (1) │        started at 2026-04-01T17:23:57.487
turbulence_closures                        (2) │   138.37 │     1.46 │   98.59 │  47.43 │ 34.3 │   13671.11 │  2187.47 │
diagnostics                                (2) │        started at 2026-04-01T17:24:24.007
diagnostics                                (2) │    30.80 │    33.28 │   98.23 │   4.83 │ 15.7 │    1963.07 │  2187.47 │
anelastic_pressure_solver_analytic         (2) │        started at 2026-04-01T17:26:03.319
parcel_dynamics                            (1) │    77.96 │    10.99 │   91.90 │  16.80 │ 21.6 │    3876.88 │  2029.50 │
moist_air_buoyancy                         (1) │        started at 2026-04-01T17:26:13.520
anelastic_pressure_solver_analytic         (2) │    26.23 │    52.17 │   97.00 │   0.00 │  0.0 │    1357.11 │  2187.47 │
gray_radiative_transfer                    (2) │        started at 2026-04-01T17:28:07.708
moist_air_buoyancy                         (1) │    17.44 │    51.16 │   98.50 │   0.00 │  0.0 │    1334.13 │  2061.86 │
polynomial_bulk_coefficients               (1) │        started at 2026-04-01T17:28:17.498
polynomial_bulk_coefficients               (1) │     2.35 │    43.42 │   45.37 │   0.00 │  0.0 │      21.79 │  2061.86 │
reference_states                           (1) │        started at 2026-04-01T17:30:03.731
gray_radiative_transfer                    (2) │    25.90 │    41.07 │   92.17 │   0.00 │  0.0 │     758.82 │  2187.47 │
tracer_dynamics                            (2) │        started at 2026-04-01T17:30:17.092
tracer_dynamics                            (2) │    10.15 │    53.32 │   96.22 │   0.00 │  0.0 │    1209.11 │  2187.47 │
advection_schemes                          (2) │        started at 2026-04-01T17:31:50.199
reference_states                           (1) │    66.64 │    65.67 │   97.57 │  18.50 │ 27.8 │    2690.26 │  2061.86 │
piecewise_linear_stretching                (1) │        started at 2026-04-01T17:32:47.770
piecewise_linear_stretching                (1) │     2.09 │    35.84 │   85.17 │   0.00 │  0.0 │     111.95 │  2061.86 │
forcing_and_boundary_conditions            (1) │        started at 2026-04-01T17:33:44.667
advection_schemes                          (2) │   122.63 │    25.61 │   97.74 │  41.29 │ 33.7 │    5705.71 │  2187.47 │
cloud_microphysics_2M                      (2) │        started at 2026-04-01T17:34:31.801
cloud_microphysics_2M                      (2) │   116.13 │    31.21 │   81.67 │  32.93 │ 28.4 │    7041.94 │  2187.47 │
dynamics                                   (2) │        started at 2026-04-01T17:37:14.642
forcing_and_boundary_conditions            (1) │   326.97 │    25.38 │   98.72 │ 121.10 │ 37.0 │   17006.34 │  2061.86 │
cloud_microphysics_0M                      (1) │        started at 2026-04-01T17:40:32.949
dynamics                                   (2) │   217.10 │    22.26 │   96.75 │  92.65 │ 42.7 │   13342.17 │  2187.47 │
cloud_microphysics_0M                      (1) │     9.65 │    38.07 │   96.44 │   0.00 │  0.0 │    1071.85 │  2329.80 │

I wonder if we're computing the memory usage wrong, seeing exactly the same RSS after multiple tests looks a bit suspicious.

[christiangnrd]

Reducing number of jobs to 2, helped a little bit, but the increasing init times are still there:

You'd probably have to remove the hard-coded number of jobs to fix the thrashing, but with those timings it's probably faster to deal with some of it than to have tests run on 1 jobs

I wonder if we're computing the memory usage wrong, seeing exactly the same RSS after multiple tests looks a bit suspicious.

Could this happen when a testset uses less memory than the previous testset but it's same worker/process so the MaxRSS value doesn't go down?

[giordano]

You'd probably have to remove the hard-coded number of jobs

Well, that was on purpose because otherwise we always get a single job, which doesn't seem very parallel. I was already setting a lower value of JULIA_TEST_MAXRSS_MB in the macOS jobs, with NumericalEarth/Breeze.jl#602 I further reduced it to 1800 MB (down from 2500 MB, but the reported RSS was never that high, causing workers to never be stopped at any point) and that seems to have brought a substantial speedup: now the macOS tests finish in less than 20 minutes, even less than 10 if we're lucky, which is a big improvement compared to timing out after 80 minutes.

Could this happen when a testset uses less memory than the previous testset but it's same worker/process so the MaxRSS value doesn't go down?

That's not what I see in the other operating systems, there the RSS is almost strictly monotonic (there are very occasional cases where it stays constant). And in general with lots of code being generated in memory in all tests (compilation is in most cases well over 90% of runtime) it seems unlikely to me that memory usage doesn't go up after several tests.

[christiangnrd]

You'd probably have to remove the hard-coded number of jobs

Well, that was on purpose because otherwise we always get a single job, which doesn't seem very parallel. I was already setting a lower value of JULIA_TEST_MAXRSS_MB in the macOS jobs, with NumericalEarth/Breeze.jl#602 I further reduced it to 1800 MB (down from 2500 MB, but the reported RSS was never that high, causing workers to never be stopped at any point) and that seems to have brought a substantial speedup: now the macOS tests finish in less than 20 minutes, even less than 10 if we're lucky, which is a big improvement compared to timing out after 80 minutes.

Glad I was wrong!

That's not what I see in the other operating systems, there the RSS is almost strictly monotonic (there are very occasional cases where it stays constant). And in general with lots of code being generated in memory in all tests (compilation is in most cases well over 90% of runtime) it seems unlikely to me that memory usage doesn't go up after several tests.

Weird. Should there be an issue to track this? (Or is this that issue?)

[giordano]

Should there be an issue to track this? (Or is this that issue?)

I guess this one can be it 😛 I'm not even sure the memory measurement is wrong, maybe macOS just does things very differently from other systems, but my experience with Breeze maybe indicates we should have a lower default memory threshold on macOS?