systemtests: add configurable timeout per test case by AdityaGupta716 · Pull Request #735 · precice/tutorials

AdityaGupta716 · 2026-03-02T10:19:55Z

Adds an optional timeout field to entries in tests.yaml, allowing per-test timeout configuration instead of a single hardcoded global value.

Closes #371

Changes

metadata_parser/metdata.py — added timeout: int = 600 to ReferenceResult
systemtests/TestSuite.py — parses optional timeout from tests.yaml; accepts both integer (600) and string (600s) formats; validates that the value is positive
systemtests/Systemtest.py — renamed GLOBAL_TIMEOUT to BUILD_TIMEOUT (docker build stays fixed); solver run and fieldcompare now use the per-test configured timeout
tools/tests/README.md — documented the new optional field with an example

Example usage in `tests.yaml`

- path: heat-exchanger
  case_combination:
    - fluid-openfoam
    - solid-calculix
  reference_result: ./heat-exchanger/reference-results/...
  timeout: 3600

Backward compatible — existing entries without timeout use the previous default of 600s.

Checklist

I added a summary of any user-facing changes in changelog-entries/371.md
I will remember to squash-and-merge

AdityaGupta716 · 2026-03-02T10:33:08Z

Hi @MakisH, the timeout is currently attached to ReferenceResult since that's where per-test configuration lives. Happy to move it to a dedicated field or separate dataclass if you prefer a different structure. Also kept BUILD_TIMEOUT fixed since build time feels infrastructure-dependent rather than simulation-dependent — let me know if you'd like that configurable too.

precice-bot · 2026-03-08T12:09:53Z

This pull request has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/gsoc-2026-aditya-gupta/2773/4

MakisH

Thanks for the PR! This is going in a good direction, but needs some further refactoring and cleanup. See some comments.

MakisH · 2026-03-26T09:06:48Z

tools/tests/metadata_parser/metdata.py

 class ReferenceResult:
    path: Path
    case_combination: CaseCombination
-


Unrelated change.

MakisH · 2026-03-26T09:07:28Z

tools/tests/systemtests/Systemtest.py



-GLOBAL_TIMEOUT = 900
+BUILD_TIMEOUT = 900


Why rename it?
How does this fit with the changes suggested/discussed in #731?

Let's discuss directly in the Changelog entry thread.

No longer renaming, GLOBAL_TIMEOUT stays as is, now overridable via PRECICE_SYSTEMTEST_TIMEOUT environment variable as suggested .

MakisH · 2026-03-26T09:09:52Z

tools/tests/components.yaml

Unrelated changes, see #731 (comment) and revert.

MakisH · 2026-03-26T09:18:34Z

tools/tests/README.md

+        timeout: 3600
 ```

+The optional `timeout` field (in seconds, default: 600) sets the maximum time allowed for the solver run and fieldcompare phases. Use this for long-running tutorials such as `heat-exchanger` or `turek-hron-fsi3`.


To be able to test this PR, you could set a timeout to one tutorial. We can remove it again before merging. For example, the Nutils cases are much slower to run than the rest, while the new DuMux cases take the longest to build.

Note that the default recently changed to 900 in #696

Added timeout: 1200 to nutils_test. Default now references to GLOBAL_TIMEOUT

MakisH · 2026-03-26T09:22:09Z

changelog-entries/735.md

@@ -0,0 +1 @@
+- Add optional `timeout` field to `tests.yaml` entries, allowing per-test timeout configuration. Defaults to 600s for backward compatibility. Applies to the solver run and fieldcompare phases.


I would not mention the default here (it recently changed to 900s in Add tutorials with DuMux and Micro-Manager into system tests #696)

Is there a design reason for only applying to solver run and fieldcompare, or is it just complicated to apply it to the build phase as well?

The total timeout now per case would be the build timeout + the case-specific timeout = 900s + 600s = 1500s = 25min by default. Is that intended? That would be much larger than what we currently have.

Removed the default mention from changelog. The per-test timeout only applies to solver run and fieldcompare because build time depends on infrastructure , not on the simulation itself. Default behavior is unchanged all three phases default to GLOBAL_TIMEOUT . The per-test timeout only takes effect when set in tests.yaml.

MakisH · 2026-03-26T09:23:36Z

tools/tests/systemtests/TestSuite.py

    name: str
    cases_of_tutorial: Dict[Tutorial, List[CaseCombination]]
    reference_results: Dict[Tutorial, List[ReferenceResult]]
+    timeouts: Dict[Tutorial, List[int]] = field(default_factory=dict)


Why is this a (dictionary with a) list per Tutorial and not a single variable per CaseCombination?

i followed it from cases_of_tutorial and reference_result thought should do the same here should i change it ?

MakisH · 2026-03-26T09:25:44Z

tools/tests/systemtests/Systemtest.py

    arguments: SystemtestArguments
    case_combination: CaseCombination
    reference_result: ReferenceResult
+    timeout: int = 600


This number seems to now be hard-coded in different places.

fixed it , now timeout: int = GLOBAL_TIMEOUT

MakisH · 2026-03-26T09:26:25Z

tools/tests/systemtests.py

            for tutorial in tutorials:
-                for case, reference_result in zip(
-                        test_suite.cases_of_tutorial[tutorial], test_suite.reference_results[tutorial]):
+                timeouts = test_suite.timeouts.get(tutorial, [600] * len(test_suite.cases_of_tutorial[tutorial]))


Why hard-coding the 600 here? Same for the generate_reference_results.py.

AdityaGupta716 force-pushed the systemtests/add-timeout-support branch from 51e92a0 to 1ffd4f0 Compare March 2, 2026 10:23

MakisH added GSoC Contributed in the context of the Google Summer of Code systemtests labels Mar 6, 2026

MakisH reviewed Mar 26, 2026

View reviewed changes

MakisH mentioned this pull request Mar 26, 2026

systemtests: add per-test max_time override for precice-config #744

Open

AdityaGupta716 mentioned this pull request Mar 26, 2026

systemtests: add heat-exchanger-simplified test suite #731

Open

2 tasks

systemtests: add configurable timeout per test case

d7345a4

AdityaGupta716 force-pushed the systemtests/add-timeout-support branch from e4b349c to d7345a4 Compare March 26, 2026 23:31

		@@ -0,0 +1 @@
		- Add optional `timeout` field to `tests.yaml` entries, allowing per-test timeout configuration. Defaults to 600s for backward compatibility. Applies to the solver run and fieldcompare phases.

Uh oh!

Conversation

AdityaGupta716 commented Mar 2, 2026

Changes

Example usage in tests.yaml

Checklist

Uh oh!

AdityaGupta716 commented Mar 2, 2026

Uh oh!

precice-bot commented Mar 8, 2026

Uh oh!

MakisH left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Example usage in `tests.yaml`