load_stac: speed up dry-run #445
Description
j-2512170901294c629b1115d305180f0a
With a lot of max-executors (30) the job now takes 2,5 hours of which one is pre-flight (assuming the stac calls as cpu usage of the driver is limited during that time)
The batch job driver starts at:
Dec 17, 2025 @ 10:09:10.663
And the post_dry_run posts its final log at (50 minutes dry-run):
Dec 17, 2025 @ 10:59:24.212
No missing x/y extent in dim_x={'type': 'spatial', 'axis': 'x', 'reference_system': {'$schema': 'https://proj.org/schemas/v0.2/projjson.schema.json', 'type': 'GeodeticCRS', 'name': 'AUTO 42001 (Universal Transverse Mercator)', 'datum': {'type': 'GeodeticReferenceFrame', 'name': 'World Geodetic System 1984', 'ellipsoid': {'name': 'WGS 84', 'semi_major_axis': 6378137, 'inverse_flattening': 298.257223563}}, 'coordinate_system': {'subtype': 'ellipsoidal', 'axis': [{'name': 'Geodetic latitude', 'abbreviation': 'Lat', 'direction': 'north', 'unit': 'degree'}, {'name': 'Geodetic longitude', 'abbreviation': 'Lon', 'direction': 'east', 'unit': 'degree'}]}, 'area': 'World', 'bbox': {'south_latitude': -90, 'west_longitude': -180, 'north_latitude': 90, 'east_longitude': 180}, 'id': {'authority': 'OGC', 'version': '1.3', 'code': 'Auto42001'}}, 'step': 10}, dim_y={'type': 'spatial', 'axis': 'y', 'reference_system': {'$schema': 'https://proj.org/schemas/v0.2/projjson.schema.json', 'type': 'GeodeticCRS', 'name': 'AUTO 42001 (Universal Transverse Mercator)', 'datum': {'type': 'GeodeticReferenceFrame', 'name': 'World Geodetic System 1984', 'ellipsoid': {'name': 'WGS 84', 'semi_major_axis': 6378137, 'inverse_flattening': 298.257223563}}, 'coordinate_system': {'subtype': 'ellipsoidal', 'axis': [{'name': 'Geodetic latitude', 'abbreviation': 'Lat', 'direction': 'north', 'unit': 'degree'}, {'name': 'Geodetic longitude', 'abbreviation': 'Lon', 'direction': 'east', 'unit': 'degree'}]}, 'area': 'World', 'bbox': {'south_latitude': -90, 'west_longitude': -180, 'north_latitude': 90, 'east_longitude': 180}, 'id': {'authority': 'OGC', 'version': '1.3', 'code': 'Auto42001'}}, 'step': 10}
Processes used in the job
adddimension1, adddimension10, adddimension11, adddimension12, adddimension13, adddimension14, adddimension15, adddimension16, adddimension17, adddimension18, adddimension19, adddimension2, adddimension20, adddimension21, adddimension22, adddimension23, adddimension24, adddimension25, adddimension26, adddimension27, adddimension3, adddimension4, adddimension5, adddimension6, adddimension7, adddimension8, adddimension9, aggregatetemporalperiod1, apply1, apply10, apply11, apply12, apply13, apply14, apply15, apply16, apply17, apply18, apply19, apply2, apply20, apply21, apply22, apply23, apply24, apply25, apply26, apply27, apply3, apply4, apply5, apply6, apply7, apply8, apply9, applydimension1, applydimension10, applydimension11, applydimension12, applydimension13, applydimension14, applydimension15, applydimension2, applydimension3, applydimension4, applydimension5, applydimension6, applydimension7, applydimension8, applydimension9, applyneighborhood1, dropdimension1, dropdimension2, dropdimension3, dropdimension4, dropdimension5, dropdimension6, dropdimension7, filterbands1, filterbands2, filterbands3, filterbands4, filterbands5, filterbands6, filterbands7, filterbbox1, filtertemporal1, filtertemporal2, loadcollection1, loadcollection2, loadstac1, mask1, mask2, mask3, mask4, mask5, mask6, mergecubes1, mergecubes10, mergecubes11, mergecubes12, mergecubes13, mergecubes14, mergecubes15, mergecubes16, mergecubes17, mergecubes18, mergecubes19, mergecubes2, mergecubes20, mergecubes21, mergecubes22, mergecubes23, mergecubes24, mergecubes25, mergecubes26, mergecubes27, mergecubes28, mergecubes29, mergecubes3, mergecubes30, mergecubes31, mergecubes32, mergecubes33, mergecubes34, mergecubes35, mergecubes36, mergecubes37, mergecubes38, mergecubes39, mergecubes4, mergecubes40, mergecubes41, mergecubes42, mergecubes43, mergecubes44, mergecubes45, mergecubes5, mergecubes6, mergecubes7, mergecubes8, mergecubes9, ndvi1, reducedimension1, reducedimension10, reducedimension11, reducedimension12, reducedimension13, reducedimension14, reducedimension15, reducedimension16, reducedimension17, reducedimension18, reducedimension19, reducedimension2, reducedimension20, reducedimension21, reducedimension22, reducedimension23, reducedimension24, reducedimension25, reducedimension26, reducedimension27, reducedimension28, reducedimension29, reducedimension3, reducedimension30, reducedimension31, reducedimension32, reducedimension33, reducedimension34, reducedimension35, reducedimension36, reducedimension37, reducedimension38, reducedimension39, reducedimension4, reducedimension40, reducedimension41, reducedimension42, reducedimension43, reducedimension5, reducedimension6, reducedimension7, reducedimension8, reducedimension9, renamelabels1, renamelabels10, renamelabels11, renamelabels12, renamelabels13, renamelabels14, renamelabels15, renamelabels16, renamelabels17, renamelabels18, renamelabels19, renamelabels2, renamelabels20, renamelabels21, renamelabels22, renamelabels23, renamelabels24, renamelabels25, renamelabels26, renamelabels27, renamelabels28, renamelabels29, renamelabels3, renamelabels30, renamelabels4, renamelabels5, renamelabels6, renamelabels7, renamelabels8, renamelabels9, resamplespatial1, resamplespatial2, resamplespatial3, saveresult1, saveresult2, saveresult3, saveresult4, saveresult5, saveresult6
"loadcollection1": {
"arguments": {
"bands": [
"B04",
"B08"
],
"id": "SENTINEL2_L2A",
"properties": {
"eo:cloud_cover": {
"process_graph": {
"lte1": {
"arguments": {
"x": {
"from_parameter": "value"
},
"y": 100
},
"process_id": "lte",
"result": true
}
}
}
},
"spatial_extent": {
"crs": "epsg:3035",
"east": 4635000,
"north": 2870000,
"south": 2855000,
"west": 4620000
},
"temporal_extent": [
"2023-01-01",
"2023-12-31"
]
},
"process_id": "load_collection"
},
"loadcollection2": {
"arguments": {
"bands": [
"SCL"
],
"id": "SENTINEL2_L2A",
"properties": {
"eo:cloud_cover": {
"process_graph": {
"lte2": {
"arguments": {
"x": {
"from_parameter": "value"
},
"y": 100
},
"process_id": "lte",
"result": true
}
}
}
},
"spatial_extent": {
"crs": "epsg:3035",
"east": 4635000,
"north": 2870000,
"south": 2855000,
"west": 4620000
},
"temporal_extent": [
"2023-01-01",
"2023-12-31"
]
},
"process_id": "load_collection"
},
"loadstac1": {
"arguments": {
"bands": [
"AGGM"
],
"temporal_extent": [
"2023-01-01",
"2023-12-31"
],
"url": "https://stac.openeo.vito.be/collections/EUGW_AGGM"
},
"process_id": "load_stac"
}