Skip to content

refactor CHIRPS3 DHIS2 workflow into modular processes#19

Open
abyot wants to merge 2 commits intomainfrom
feat/chirps3-workflow-modular-refactor
Open

refactor CHIRPS3 DHIS2 workflow into modular processes#19
abyot wants to merge 2 commits intomainfrom
feat/chirps3-workflow-modular-refactor

Conversation

@abyot
Copy link
Member

@abyot abyot commented Feb 27, 2026

Summary

Refactors the CHIRPS3 -> DHIS2 flow from a monolithic processor into modular, reusable OGC processes with a workflow entrypoint, plus safer cache behavior and cleaner naming.

What changed

  • Replaced legacy chirps3-dhis2-pipeline with a process-first design:
    • chirps3-dhis2-workflow (assembly/orchestration)
    • feature-fetch
    • chirps3-download (reused)
    • data-aggregate
    • dhis2-datavalue-build
  • Kept orchestration concerns in chirps3_workflow.py + workflow_runtime.py (run_process_with_trace).
  • Made decomposed processes self-contained.
  • Added flavor support end-to-end (rnl/sat) with stage/flavor validation.
  • Removed deprecated chirps3_dhis2_pipeline.py.
  • Updated pygeoapi process wiring and docs/diagrams accordingly.

Caching improvements

  • Added shared cache utilities (src/eo_api/utils/cache.py).
  • Improved CHIRPS download cache scoping by spatial+product dimensions (bbox, stage, flavor) to avoid stale reuse.
  • Added delta-friendly reuse:
    • download layer reuses existing monthly files and fetches only missing months
    • aggregation layer reuses cached (orgUnit, period) rows and computes only missing spatial/temporal deltas

@abyot abyot requested review from mortenoh and turban February 27, 2026 09:26
@abyot abyot marked this pull request as ready for review February 27, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant