Raster data registry, cached reading, custom api endpoints#3
Merged
karimbahgat merged 29 commits intomainfrom Feb 26, 2026
Merged
Raster data registry, cached reading, custom api endpoints#3karimbahgat merged 29 commits intomainfrom
karimbahgat merged 29 commits intomainfrom
Conversation
added 22 commits
February 23, 2026 13:02
… such as coverage, more
…eneath dataset endpoint
…ect period format
…al code so we always work with an xarray dataset, dask compute after time aggregation to avoid very slow dask graphs later
…s zarr file if it exists which is much faster
…d time for large countries like brazil
karimbahgat
changed the title
karimbahgat
changed the title
karimbahgat
changed the title
added 5 commits
February 25, 2026 15:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Self contained
datasetsmodule with its own route, independent of other parts of the repo.I dont want to optimize this code since we dont know what will or wont be used, so I tried to exclude from ruff and mypy. Still getting errors, but think we should merge and fix it later.
The core is a dataset registry/cache system based purely on yaml files, no source-specific code. Everything is generic.
To download data for a dataset, trigger
/datasets/ID/build_cache.To optimize downloaded data into efficient zarr store, trigger
datasets/ID/optimize_cache.After that you can calculate dataset stats for DHIS2 org units via
datasets/ID/PERIOD_TYPE/orgunits. To render a quick map image of what it would look like, add/previewat the end.To dynamically create time aggregated raster dataset you can use
datasets/ID/PERIOD_TYPE/raster. To render a quick map image of what it would look like, add/previewat the end.The
/tilesendpoint is just a dummy and does not work, but should be possible to just redirect to TiTiler endpoint pointing to dynamically generated file.