Feat/alert system #2599

sandeshit · 2025-11-28T10:58:11Z

Changes

Alert-system app
Polling, ETL

Checklist

Things that should succeed before merging.

Updated/ran unit tests
Updated CHANGELOG.md

Release

If there is a version update, make sure to tag the repository with the latest version.

main/sentry.py

deploy/helm/ifrcgo-helm/values.yaml

alert_system/admin.py

alert_system/models.py

alert_system/management/commands/poll_gdacs_flood.py

susilnem

Minor changes...

alert_system/tasks.py

alert_system/etl/base/extraction.py

alert_system/models.py

susilnem

Small changes!

alert_system/models.py

alert_system/tasks.py

alert_system/etl/base/extraction.py

sudip-khanal

Minor changes

alert_system/management/commands/poll_gdacs_cyclone.py

alert_system/management/commands/poll_gdacs_flood.py

alert_system/management/commands/poll_usgs_earthquake.py

susilnem

Do we have any analytics on usage and memory consumption?
Has any profiling been done so far?

alert_system/tasks.py

susilnem · 2025-12-30T05:16:49Z

alert_system/tasks.py

+        go_event_ids = list(
+            Event.objects.filter(field_reports__num_affected__gte=item.total_people_exposed, dtype=connector.dtype)
+            .values_list("id", flat=True)
+            .distinct()
+        )


thenav56 · 2025-12-30T09:18:35Z

alert_system/tasks.py

+            .distinct()
+        )
+
+        item.related_go_events = go_event_ids


Check if we have instead use M2M

Use bulk insert to add M2M (Use ignore existing)

alert_system/tasks.py

alert_system/models.py

alert_system/management/commands/poll_usgs_earthquake.py

thenav56 · 2025-12-30T09:40:41Z

alert_system/etl/base/loader.py

+        )
+
+        action = "Created" if created else "Updated"
+        logger.info(f"{action} Event for correlation_id={correlation_id}")


Suggested change

logger.info(f"{action} Event for correlation_id={correlation_id}")

logger.info(f"{action} Event for {correlation_id=}")

alert_system/etl/gdacs_flood/extraction.py

alert_system/etl/base/extraction.py

thenav56 · 2025-12-30T09:49:34Z

alert_system/etl/Gdacs_flood/extraction.py

+    event_collection_type = gdacs_flood_config.event_collection_type
+    hazard_collection_type = getattr(gdacs_flood_config, "hazard_collection_type", None)
+    impact_collection_type = getattr(gdacs_flood_config, "impact_collection_type", None)
+    filter_event = getattr(gdacs_flood_config, "filter_event", None)
+    transformer_class = GdacsTransformer
+    loader_class = GdacsLoader


We can replace all this with a single typed dict...

ranjan-stha · 2026-01-08T04:41:56Z

alert_system/etl/base/extraction.py

+    def _hazard_filter(self, unit: str, value: int) -> str:
+        return f"monty:hazard_detail.severity_unit = '{unit}' AND " f"monty:hazard_detail.severity_value >= {value}"


@sandeshit To verify: unit and value mayn't exist in all the source items. Handle it if it is None or NaN.

alert_system/etl/base/extraction.py

ranjan-stha · 2026-01-08T04:54:31Z

alert_system/etl/gdacs_flood/config.py

+    "event_collection_type": "gdacs-events",
+    "hazard_collection_type": "gdacs-hazards",
+    "impact_collection_type": "gdacs-impacts",
+    "filter_event": {"hazard_codes": ["FL", "MH0600", "nat-hyd-flo-flo"]},


@sandeshit can we handle multiple set of same/similar hazards? Example: FL is a general one for Flood but there are other types of Flood as well like Flash Flood etc. What if we want to handle different sets of similar hazard types. Might need to discuss.

alert_system/etl/gdacs_flood/transform.py

alert_system/etl/usgs_earthquake/transform.py

- Nested extraction with correlation id as fliter. - Self retry with exponential backoff. - Different models for Hazards and Impacts.

This commit splits the data extraction, transform and load part. The filtration part is also included in the load part.

- Added category and type enum. - Alter people and building exposed fallback to None. - Add polling start date and lookback weeks field in Connector. - Squash Migrations.

susilnem · 2026-01-20T05:58:50Z

alert_system/models.py

+    class Meta:
+        verbose_name = _("Connector")
+        verbose_name_plural = _("Connectors")
+        ordering = ["type"]


Suggested change

ordering = ["type"]

ordering = ["id","type"]

susilnem · 2026-01-20T06:02:05Z

alert_system/helpers.py

+    connector.save(update_fields=["status", "last_success_run"] if status == Connector.Status.SUCCESS else ["status"])
+
+
+def get_connector_processor(connector_id: int):


Move this section to model.py

susilnem · 2026-01-20T06:02:45Z

alert_system/helpers.py

+from alert_system.models import Connector
+
+
+def set_connector_status(connector, status):


This one too

susilnem · 2026-01-20T06:03:05Z

alert_system/tasks.py

+    processor, connector = get_connector_processor(connector_id)
+
+    try:
+        set_connector_status(connector, Connector.Status.RUNNING)


Suggested change

set_connector_status(connector, Connector.Status.RUNNING)

connector.set_connector_status(Connector.Status.RUNNING)

susilnem · 2026-01-20T06:50:07Z

alert_system/management/commands/poll_gdacs_cyclone.py

+    SOURCE_TYPE = Connector.ConnectorType.GDACS_CYCLONE
+
+    def handle(self, *args, **options):
+        self.stdout.write("Starting extraction task...")


Suggested change

self.stdout.write("Starting extraction task...")

self.stdout.write("Starting extraction task for SOURCE_TYPE.label ...")

- Remove mappings.py. - Move connector registry inside connector model. - Rename variables in tasks.py.

sandeshit mentioned this pull request Nov 28, 2025

Alert-system polling #2584

Closed

2 tasks

sandeshit requested review from susilnem and thenav56 December 2, 2025 04:46

susilnem requested changes Dec 5, 2025

View reviewed changes

sandeshit force-pushed the feat/alert-system branch from 3fa8ab2 to 9fd5372 Compare December 7, 2025 17:49

sandeshit changed the base branch from project/alert-system-workflow to develop December 7, 2025 18:22

sandeshit changed the base branch from develop to project/alert-system-workflow December 7, 2025 18:23

sandeshit marked this pull request as draft December 7, 2025 18:24

sandeshit requested a review from susilnem December 8, 2025 09:09

susilnem requested changes Dec 9, 2025

View reviewed changes

susilnem requested changes Dec 11, 2025

View reviewed changes

sudip-khanal requested changes Dec 11, 2025

View reviewed changes

alert_system/management/commands/poll_gdacs_cyclone.py Outdated Show resolved Hide resolved

alert_system/management/commands/poll_gdacs_flood.py Outdated Show resolved Hide resolved

alert_system/management/commands/poll_usgs_earthquake.py Outdated Show resolved Hide resolved

sandeshit requested a review from susilnem December 12, 2025 09:02

susilnem requested changes Dec 30, 2025

View reviewed changes

thenav56 requested changes Dec 30, 2025

View reviewed changes

sandeshit force-pushed the feat/alert-system branch from c4d7b89 to fcc9bdd Compare January 2, 2026 08:42

ranjan-stha reviewed Jan 8, 2026

View reviewed changes

sandeshit force-pushed the project/alert-system-workflow branch from d71e1d0 to 7dc5624 Compare January 19, 2026 06:27

sandeshit added 12 commits January 19, 2026 12:16

chore(app): base setup for alert system app

5f894b2

feat(polling): extraction logic for polling

4f35aaf

chore(sentry): add cron monitor for tracking

c599bb1

feat(commands): management commands to run different polling tasks

7e6679d

refactor(extraction): refactor extraction logic for hazards and impacts.

495366d

- Nested extraction with correlation id as fliter. - Self retry with exponential backoff. - Different models for Hazards and Impacts.

feat(extraction): use different classes for different sources

6e9b2e3

feat(filter): add filtration classes and cronjob

1bd780e

feat(etl): refactor existing extraction into ETL.

ccb69f4

This commit splits the data extraction, transform and load part. The filtration part is also included in the load part.

feat(etl): add past events fetcher

47edeb9

chore(alert-system): clean up naming and cronjobs

64ddf37

feat(etl): add past events from GO event table

5428797

feat(etl): add usgs source

ea937c9

sandeshit added 3 commits January 19, 2026 12:16

chore(etl): separate past event extraction into different class

2ae64fa

fixup! chore(etl): separate past event extraction into different class

0d15b14

feat(etl): change transform logic.

13121a5

- Added category and type enum. - Alter people and building exposed fallback to None. - Add polling start date and lookback weeks field in Connector. - Squash Migrations.

sandeshit force-pushed the feat/alert-system branch from ad0aa8c to 13121a5 Compare January 19, 2026 08:43

sudip-khanal and others added 2 commits January 19, 2026 15:38

chore(assets): point to latest open-api schema

a28c4b0

chore(schema): fix ci issues in schema

0c79300

susilnem requested changes Jan 20, 2026

View reviewed changes

chore(models): move mappings inside the connector model.

93e53d5

- Remove mappings.py. - Move connector registry inside connector model. - Rename variables in tasks.py.

	logger.info(f"{action} Event for correlation_id={correlation_id}")
	logger.info(f"{action} Event for {correlation_id=}")

		def _hazard_filter(self, unit: str, value: int) -> str:
		return f"monty:hazard_detail.severity_unit = '{unit}' AND " f"monty:hazard_detail.severity_value >= {value}"

		connector.save(update_fields=["status", "last_success_run"] if status == Connector.Status.SUCCESS else ["status"])


		def get_connector_processor(connector_id: int):

		from alert_system.models import Connector


		def set_connector_status(connector, status):

	set_connector_status(connector, Connector.Status.RUNNING)
	connector.set_connector_status(Connector.Status.RUNNING)

	self.stdout.write("Starting extraction task...")
	self.stdout.write("Starting extraction task for SOURCE_TYPE.label ...")

Feat/alert system #2599

Are you sure you want to change the base?

Feat/alert system #2599

Uh oh!

Conversation

sandeshit commented Nov 28, 2025

Changes

Checklist

Release

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

susilnem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

susilnem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sudip-khanal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

susilnem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment