Smart Overpass fetching, SQLite node cache, and rate-limit handling by dougborg · Pull Request #133 · FoggedLens/deflock-app

dougborg · 2026-02-26T16:55:57Z

Summary

Semaphore-managed Overpass requests: Limits concurrent requests to 2 (matching Overpass slot count per IP), with priority queueing so user-initiated requests always jump ahead of background prefetch/refresh work
Generation-based cancellation: Stale fetch requests are cancelled at multiple checkpoints when the user pans away, without aborting in-flight HTTP calls (completed responses are still cached)
Pre-flight rate-limit detection: Checks /api/status before expensive POST requests; semaphore cooldown pauses all requests for the server-reported wait time, with single-slot recovery to avoid burning both slots simultaneously
SQLite write-through persistence: Node cache survives app restarts with 7-day TTL, automatic pruning of expired areas on startup, and orphaned node cleanup
Progressive rendering: Each completed quadrant updates the map immediately via throttled notifications (200ms batching)
Coverage overlay: "Fog of war" visualization showing which areas have loaded data, with age-colored debug tinting (green/yellow/orange) in dev mode
Rate-limit countdown UI: Circular countdown in the network status indicator showing server-reported wait time draining in real time
Settle-before-fetch: Defers Overpass requests until the map settles (no fingers down, fling animation done) to avoid burning slots on immediately-stale requests

How cancellation works

_fetchGeneration increments when the user pans to an uncached area. All in-flight work snapshots this value and bails at checkpoints when stale:

Before entering the semaphore queue
After waking from the queue, before making the HTTP call
Before spawning split sub-requests (NodeLimitError)
Before starting the split-fetch loop
Before each quadrant in a split fetch

Additionally, stale-but-completed HTTP responses are still cached (the nodes.isNotEmpty guard ensures real data from a stale generation isn't thrown away). Post-fetch result handling in getNodesFor also checks staleness to skip UI updates for superseded requests.

Once an HTTP call is in flight, it can't be cancelled (Dart limitation). What generation checks prevent is new work from starting: queued requests, split sub-requests, and rate-limit retries.

_prefetchGeneration increments when the user pans to a cached area (triggering a new prefetch) or when a rate limit sets cooldown. This cancels the previous prefetch loop without affecting user-initiated fetches.

Rate limiting

Pre-flight /api/status check → if no slot available, throw RateLimitError with server-reported wait time (+2s buffer) → semaphore enters cooldown (all requests pause) → reduce to 1 slot to prevent two concurrent 7s queries from burning both slots → reconciliation timer fires after cooldown + 1s buffer → retry viewport fetch → on success, restore to 2 slots.

Background refreshes and prefetches do not retry on 429 — they bail immediately so slots serve the user.

Other design decisions

Retries and splits re-enter the semaphore: NodeLimitError handlers run outside the semaphore (slot already released), competing fairly in the FIFO queue
Cache compaction: markAreaAsFetched prunes in-memory entries subsumed by the new area; insertCachedArea does the same in SQLite; deleteExpiredData uses NOT EXISTS subquery to avoid SQLite's 999 bind-variable limit
Targeted refresh: refreshArea (manual retry) invalidates only overlapping cached areas instead of clearing the entire cache
Node render prioritization: Sorts by distance from viewport center when render limit is active, so visible nodes always make the cut
In-flight dedup: When an in-flight request's expanded bounds already cover the new viewport, the new fetch is skipped — the 1.2x expansion (1.5x at low zoom) usually covers small pans

Files changed

File	Change
`lib/services/node_data_manager.dart`	Semaphore with cooldown/restore, generation counter, split fetching, background refresh, prefetching, reconciliation
`lib/services/node_cache_database.dart`	New — SQLite persistence with batch upsert, TTL, orphan cleanup
`lib/services/node_spatial_cache.dart`	`initPersistence()`, `hasFreshDataFor()`, `staleAreaFor()`, `invalidateArea()`, write-through, area compaction, cached `fetchedAreas` view
`lib/services/overpass_service.dart`	Pre-flight `/api/status` check via `secondsUntilSlotAvailable()`, `RateLimitError` with `waitSeconds`, shared `_parseWaitSeconds` helper
`lib/services/network_status.dart`	`setRateLimited(waitSeconds:)`, dynamic auto-reset timer using actual wait time
`lib/widgets/network_status_indicator.dart`	`StatefulWidget` with rate-limit countdown circle and proportional progress updates
`lib/widgets/map/coverage_overlay.dart`	New — fog-of-war polygon layer with age-colored debug tinting
`lib/widgets/map/map_data_manager.dart`	Distance-based node prioritization when render limit active
`lib/widgets/map_view.dart`	Settle-before-fetch, pointer-up fetch scheduling, coverage overlay integration, timer disposal
`lib/widgets/map/node_refresh_controller.dart`	Skip cache clear on initial profile set (preserve SQLite hydration)
`lib/widgets/node_provider_with_cache.dart`	Listen to `NodeDataManager` for progressive updates, dispose cleanup
`lib/app_state.dart`	Wire `initPersistence()` before `preloadOfflineNodes()`, coverage overlay setting
`lib/state/settings_state.dart`	Coverage overlay toggle persistence
`lib/screens/advanced_settings_screen.dart`	Re-enable network status section with coverage toggle
`lib/services/map_data_provider.dart`	Document download semaphore behavior
`pubspec.yaml`	SDK constraint `>=3.10.0`, add `sqflite_common_ffi` dev dep
`test/services/node_data_manager_test.dart`	New — 24 tests
`test/services/node_cache_database_test.dart`	New — 17 tests
`test/widgets/coverage_overlay_test.dart`	New — 3 tests

Test plan

flutter test — all tests pass, no regressions
flutter analyze — no issues
Dense area at zoom 10-11 → cameras appear progressively as quadrants complete
Kill and relaunch app → cached nodes reappear instantly from SQLite
Pan around cached area → prefetch cells populate surrounding areas
Trigger rate limit → countdown circle shows, reconciliation retries after cooldown
Coverage overlay toggle in settings shows/hides fog-of-war

🤖 Generated with Claude Code

Copilot

Pull request overview

Enables progressive map updates while fetching Overpass quadrants in parallel by propagating mid-fetch NodeDataManager cache updates through NodeProviderWithCache, and adds unit tests for the per-quadrant notification behavior.

Changes:

Emit NodeDataManager.notifyListeners() as each non-empty quadrant result is cached, enabling progressive rendering.
Have NodeProviderWithCache subscribe to NodeDataManager updates (and remove the manual post-fetch provider notification).
Add tests (and fake_async) to verify per-quadrant notification behavior and Overpass status polling behavior.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`lib/services/node_data_manager.dart`	Adds per-quadrant `notifyListeners()` after caching and threads stale-generation + semaphore logic through split fetches.
`lib/widgets/node_provider_with_cache.dart`	Subscribes provider to `NodeDataManager` updates; removes redundant provider-side notify after fetch.
`lib/services/overpass_service.dart`	Adds `/api/status` slot-count parsing and polling wait helper for rate limiting.
`lib/services/node_spatial_cache.dart`	Adds a `forTesting` constructor to allow non-singleton cache instances in tests.
`lib/services/map_data_provider.dart`	Documents that offline downloads pass null generation (not stale-cancelled) and may occupy semaphore slots.
`test/services/node_data_manager_test.dart`	Adds extensive tests including progressive rendering notification counts.
`pubspec.yaml`	Adds `fake_async` as a direct dev dependency.
`pubspec.lock`	Updates dependency metadata for `fake_async` to direct dev.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

lib/widgets/node_provider_with_cache.dart

dougborg · 2026-02-26T22:08:53Z

This one could merge in place of #114 - it has all those commits too.

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

lib/services/node_data_manager.dart

lib/services/node_spatial_cache.dart

lib/services/map_data_provider.dart

lib/services/node_cache_database.dart

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

lib/widgets/map_view.dart

lib/services/network_status.dart

lib/widgets/network_status_indicator.dart

lib/services/overpass_service.dart

lib/services/node_cache_database.dart

lib/screens/settings/sections/network_status_section.dart

Copilot

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

lib/services/node_spatial_cache.dart

lib/widgets/network_status_indicator.dart

lib/services/node_data_manager.dart

lib/services/node_spatial_cache.dart

Replace fire-and-forget Overpass requests with a managed pipeline: - Semaphore limits concurrent requests to the Overpass slot count (2), with priority queueing so user-initiated requests jump ahead of background work - Generation counter cancels stale requests at 6 checkpoints when the user pans away, without aborting in-flight HTTP calls - Automatic area splitting on NodeLimitError (up to 4^3 = 64 sub-areas) with progressive rendering as each quadrant completes - Pre-flight /api/status check avoids burning time on requests that will be rejected; semaphore cooldown pauses all requests during rate limits with server-reported wait times - Reconciliation timer auto-retries failed viewport fetches after cooldown expires - SQLite write-through persistence (7-day TTL) so the node cache survives app restarts; expired areas pruned on startup - Background refresh of stale cached areas (>1hr) targeting original bounds; ring-based prefetching of surrounding areas at higher zooms - Defer fetches until map settles (no fingers down, fling done) to avoid burning Overpass slots on immediately-stale requests - Skip redundant fetches when in-flight request covers the viewport Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

lib/widgets/network_status_indicator.dart

lib/services/node_data_manager.dart

- Coverage overlay ("fog of war") shows which map areas have loaded data, with age-colored debug tinting in dev mode (green/yellow/orange) - Rate-limit countdown circle in the network status indicator shows server-reported wait time draining in real time; proportional progress updates prevent timer jumps on repeated rate-limit reports - Prioritize closest nodes to viewport center when render limit is active, preventing visible gaps that shift as you pan - Add coverage overlay toggle in advanced settings - Cancel constrained-node snap-back timer in dispose Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… math Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

dougborg · 2026-03-09T20:55:32Z

lib/widgets/map/map_data_manager.dart

-      // Apply rendering limit to prevent UI lag
+      // Apply rendering limit to prevent UI lag.
+      // Sort by distance from viewport center so the most visible nodes
+      // always make the cut, preventing gaps that shift as you pan.
      if (validNodes.length > maxNodes) {
+        final centerLat = (mapBounds!.north + mapBounds.south) / 2;
+        final centerLng = (mapBounds.east + mapBounds.west) / 2;
+        validNodes.sort((a, b) {
+          final distA = (a.coord.latitude - centerLat) * (a.coord.latitude - centerLat) +
+                        (a.coord.longitude - centerLng) * (a.coord.longitude - centerLng);
+          final distB = (b.coord.latitude - centerLat) * (b.coord.latitude - centerLat) +
+                        (b.coord.longitude - centerLng) * (b.coord.longitude - centerLng);
+          return distA.compareTo(distB);
+        });


This is a small improvement that massively improves the UX imo. I think we should do this no matter what else happens with the other changes in the PR.

stopflock · 2026-03-13T16:46:40Z

Can you rebase this? I want to play with a build on top of latest

Copilot AI review requested due to automatic review settings February 26, 2026 16:55

Copilot started reviewing on behalf of dougborg February 26, 2026 16:56 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

lib/widgets/node_provider_with_cache.dart Show resolved Hide resolved

dougborg mentioned this pull request Mar 7, 2026

Parallelize quadrant fetching and add smart 429 retry #114

Closed

5 tasks

dougborg force-pushed the feat/progressive-quadrant-rendering branch 3 times, most recently from a63e21d to e1b5c0a Compare March 7, 2026 22:49

dougborg changed the title ~~Progressive rendering during parallel quadrant fetching~~ Smart Overpass fetching, SQLite node cache, and background prefetching Mar 8, 2026

dougborg force-pushed the feat/progressive-quadrant-rendering branch 4 times, most recently from 642bdec to a63a87a Compare March 8, 2026 07:07

dougborg requested a review from Copilot March 8, 2026 07:12

Copilot started reviewing on behalf of dougborg March 8, 2026 07:12 View session