Add custom CMOR table and derivation script for permafrost #2358

axel-lauer · 2024-03-04T08:48:54Z

Description

This PR adds (a) custom CMOR tables for the following three ESACCI-PERMAFROST variables:

active layer thickness (alt)
permafrost ground temperature (gtd)
permafrost extent (pfr)

and (b) a derivation script for permafrost extent (pfr) to calculate this variable from the CMIP models.

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🧪 The new functionality is relevant and scientifically sound
🛠 This pull request has a descriptive title and labels
🛠 Code is written according to the code quality guidelines
~~- [ ] 🧪 and 🛠 Documentation is available~~
🛠 Unit tests have been added
🛠 Changes are backward compatible
~~- [ ] 🛠 Any changed dependencies have been added or removed correctly~~
~~- [ ] 🛠 The list of authors is up to date~~
🛠 All checks below this pull request were successful

codecov · 2024-03-04T08:55:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.61%. Comparing base (9244717) to head (c5453bb).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2358      +/-   ##
==========================================
+ Coverage   95.59%   95.61%   +0.01%     
==========================================
  Files         266      267       +1     
  Lines       15573    15629      +56     
==========================================
+ Hits        14887    14943      +56     
  Misses        686      686

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-06-27T03:10:34Z

In order to maintain a backlog of relevant pull requests, we automatically label them as stale after 180 days of inactivity.

If this pull request is still important to you, please comment below to remove the stale label. Otherwise, this pull request will be automatically closed in 60 days. If this pull request only suffers from a lack of reviewers, please tag the @ESMValGroup/technical-lead-development-team so they can help you find a suitable reviewer.

bouweandela

It looks like there is also a derivation function in this pull request that is not mentioned in the description at the top. Is that intentional? If yes, please make sure that the derivation is lazy and add a unit test for it.

esmvalcore/preprocessor/_derive/pfr.py

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

esmvalcore/preprocessor/_derive/pfr.py

bouweandela · 2026-01-12T15:37:24Z

If you click the Update branch button or run git pull origin main --no-ff and git push locally, the tests on CircleCI should run again.

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

… into esacci-permafrost

esmvalcore/preprocessor/_derive/pfr.py

bouweandela · 2026-01-14T13:13:19Z

esmvalcore/preprocessor/_derive/pfr.py

+        # loop over all years and test if frost is present throughout
+        # the whole test period, i.e. [year-test_period, year+test_period]
+
+        for tidx, year in enumerate(year_coord.points):
+            # extract test period
+            pdt1 = PartialDateTime(
+                year=year - 1,
+                month=13 - test_period,
+                day=1,
+            )
+            pdt2 = PartialDateTime(year=year + 1, month=test_period + 1, day=1)
+            daterange = iris.Constraint(
+                time=lambda cell, pdt1=pdt1, pdt2=pdt2: pdt1
+                <= cell.point
+                < pdt2,
+            )
+            soiltemp_window = soiltemp.extract(daterange)
+            # remove auxiliary coordinate 'year' to avoid lots of warnings
+            # from iris
+            soiltemp_window.remove_coord("year")
+            # calculate mean over test period
+            test_cube = soiltemp_window.collapsed("time", iris.analysis.MEAN)
+            # if all months in test period show soil tempeatures below zero
+            # then mark grid cell with "1" as permafrost and "0" otherwise
+            pfr_yr.data[tidx, :, :] = da.where(test_cube.data > 0.99, 1, 0)


Could this be replaced by the rolling_window_statistics preprocessor function? Something like:

pfr_yr = esmvalcore.preprocessor.rolling_window_statistics( soiltemp, coordinate="time", operator="mean", window_length=FROZEN_MONTHS, )

Not sure. I tried a few options but could not reproduce the original output. Not sure it is worth spending much more time on this possible replacement.

I rolling_window_statistics could be used if you extend the input cube by test_period time steps to both sides (the rolling window with a window > 1 will produce a shorter output cube; this accounts for that). Not sure if that's makes the code simpler, probably not...

It looks like the current implementation creates fairly small dask chunks, which may lead to slow/inefficient computations. Every time you create a dask array and use it to build another dask array, you insert all the dask chunks needed to build that subarray into the final array. Looping over the time-axis and computing things separately for each time step is a typical example of this. This may be aggravated by the use of aggregated_by SciTools/iris#5455. Therefore, I think it would be worth investigating if rolling_window_statistics can be used to reduce the size of the Dask task graph and increase the chunk sizes.

However, maybe this works just fine, it's hard to tell from just looking at the code. Did you test the runtime? Can you run the computation using the distributed scheduler? And did you test how long it takes to compute a multi-model mean over e.g. 100 datasets?

It would be nice to have a good unit test, so we can look at improving computational performance later https://github.com/ESMValGroup/ESMValCore/pull/2358/files#r2690416067 if needed.

tests/unit/preprocessor/_derive/test_pfr.py

bouweandela · 2026-01-14T13:22:51Z

tests/unit/preprocessor/_derive/test_pfr.py

+    out_cube = derived_var.calculate(in_cubes)
+    assert out_cube.units == cf_units.Unit("%")
+    out_data = out_cube.data
+    expected = 100.0 * np.ones_like(out_cube.data)


It would be nice to have a test case where not the entire cube is permafrost because this doesn't really check that the calculation is correct.

esmvalcore/preprocessor/_derive/pfr.py

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

schlunma · 2026-01-20T09:55:03Z

Hello, this pull request has been marked with the v2.14.0 milestone. The release of version 2.14.0 is currently scheduled for February 2026. To get this into the new release, it would be great to get this merged by the end of January.

If you won't be able to finish this in time, don't worry - just unassign the milestone v2.14.0. If you need any support, ping myself (@schlunma; the release manager for v2.14.0) or the @ESMValGroup/technical-lead-development-team. Please note that I won't be available until the beginning of February, though.

esmvalcore/preprocessor/_derive/pfr.py

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

codecov · 2026-01-22T10:00:53Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.61%. Comparing base (9244717) to head (6913302).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2358      +/-   ##
==========================================
+ Coverage   95.59%   95.61%   +0.01%     
==========================================
  Files         266      267       +1     
  Lines       15573    15629      +56     
==========================================
+ Hits        14887    14943      +56     
  Misses        686      686

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

axel-lauer added 2 commits February 28, 2024 10:25

added custom CMOR tables for ESACCI-PERMAFROST

e019ab5

added new coordinate to custom coordinate table

fbce425

axel-lauer added cmor Related to the CMOR standard observations labels Mar 4, 2024

axel-lauer mentioned this pull request Mar 4, 2024

CMORizer ESACCI-PERMAFROST ESMValGroup/ESMValTool#3545

Open

11 tasks

changed sdepth to depth

cc102fa

axel-lauer changed the title ~~Add custom CMO tables for ESACCI-PERMAFROST~~ Add custom CMOR tables for ESACCI-PERMAFROST Mar 4, 2024

axel-lauer added 3 commits June 12, 2024 07:59

Merge branch 'main' into esacci-permafrost

c38bda3

added derivation script for permafrost extent (pfr)

659494d

permafrost extent: switched from sftgif to mrsos

b4f46b9

github-actions bot added the Stale label Jun 27, 2025

axel-lauer added 2 commits July 23, 2025 11:46

Merge branch 'main' into esacci-permafrost

0c86835

updated pfr.py

4db4a42

github-actions bot removed the Stale label Jul 24, 2025

axel-lauer marked this pull request as ready for review July 25, 2025 11:19

bouweandela reviewed Dec 5, 2025

View reviewed changes

esmvalcore/preprocessor/_derive/pfr.py Outdated Show resolved Hide resolved

axel-lauer and others added 8 commits December 19, 2025 09:52

Update esmvalcore/preprocessor/_derive/pfr.py

9ee7825

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

fix ruff issues

480d33d

update pfr.py

26c19da

removed comments

e562830

Merge branch 'main' into esacci-permafrost

f4fb071

non-working draft of pfr derivation unit test

b616d92

added test for derived variable pfr

8a92845

pre-commit results

1db80de

axel-lauer changed the title ~~Add custom CMOR tables for ESACCI-PERMAFROST~~ Add custom CMOR table and derivation script for permafrost Jan 9, 2026

axel-lauer added this to the v2.14.0 milestone Jan 12, 2026

bouweandela reviewed Jan 12, 2026

View reviewed changes

esmvalcore/preprocessor/_derive/pfr.py Outdated Show resolved Hide resolved

axel-lauer and others added 5 commits January 13, 2026 08:19

Update esmvalcore/preprocessor/_derive/pfr.py

5d0f518

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

Merge branch 'main' into esacci-permafrost

66eb70a

extended unit test for derivation of pfr

b3d0fca

Merge branch 'esacci-permafrost' of github.com:ESMValGroup/ESMValCore…

3177cd8

… into esacci-permafrost

update test_pfr.py

d370a3b

bouweandela requested changes Jan 14, 2026

View reviewed changes

axel-lauer and others added 6 commits January 15, 2026 09:03

Update esmvalcore/preprocessor/_derive/pfr.py

01c8227

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

Update esmvalcore/preprocessor/_derive/pfr.py

cce9088

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

Update esmvalcore/preprocessor/_derive/pfr.py

59bc9c0

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

split unit tests into separate functions

ee34de9

updated pfr

f20e3d7

Merge branch 'main' into esacci-permafrost

c5453bb

bouweandela reviewed Jan 22, 2026

View reviewed changes

esmvalcore/preprocessor/_derive/pfr.py Outdated Show resolved Hide resolved

Update esmvalcore/preprocessor/_derive/pfr.py

6913302

Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>

Add custom CMOR table and derivation script for permafrost #2358

Are you sure you want to change the base?

Add custom CMOR table and derivation script for permafrost #2358

Uh oh!

Conversation

axel-lauer commented Mar 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

codecov bot commented Mar 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jun 27, 2025

Uh oh!

bouweandela left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bouweandela commented Jan 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bouweandela Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

axel-lauer Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

schlunma Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

bouweandela Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bouweandela Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

schlunma commented Jan 20, 2026

Uh oh!

Uh oh!

codecov bot commented Jan 22, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

axel-lauer commented Mar 4, 2024 •

edited

Loading

codecov bot commented Mar 4, 2024 •

edited

Loading