Add Output Area crosswalk and geographic assignment (Phase 1)

[vahid-ahmadi]

Background

This PR implements Phase 1 of a 6-phase pipeline to enable Output Area (OA) level calibration — the UK equivalent of the US Census Block approach.

Why are we doing this?

The US pipeline (policyengine-us-data) uses a clone-and-prune approach that produces much finer geographic granularity than our current UK methodology:

Aspect	UK (current)	US (target approach)
Finest geography	Constituency (650) / LA (360)	Census Block (~8M)
Method	PyTorch Adam reweighting — every household keeps some weight	L0-regularized optimization — actively drops records for sparsity
Cloning	None — each FRS household appears once	Records cloned 10-430x, each clone assigned different geography
Weight matrix	Dense (n_areas × n_households)	Sparse (most clones pruned to zero)

This PR is going down to Output Area level (~235K OAs across the UK), which is the UK equivalent of the US Census Block. This PR is the first step.

What this PR does (Phase 1: OA Crosswalk & Geographic Assignment)

1. Unified UK Output Area Crosswalk

Downloads and combines geographic lookups from three national statistics agencies into a single crosswalk:

```
OA → LSOA/DataZone → MSOA/IntermediateZone → LA → Constituency → Region → Country
```

Data sources:

• England & Wales (188,880 OAs): ONS Open Geography Portal — OA21→LSOA21→MSOA21→LAD22 exact-fit lookup, OA21→PCON25 constituency lookup, Census 2021 TS001 population
• Scotland (46,363 OAs): NRS OA22→DZ22→IZ22 lookup, statistics.gov.scot DZ22→LA→constituency lookup, OA22→UKPC24 direct constituency mapping
• Northern Ireland: NISRA DZ2021 lookup (currently returning 404 — gracefully skipped, to be added when URLs are updated)

Output: `storage/oa_crosswalk.csv.gz` (1.4MB compressed) — 235,243 areas, 65M population, 632 constituencies, 363 LAs, 11 regions

2. Geographic Assignment Engine

Assigns population-weighted random Output Areas to cloned FRS household records, with two key constraints:

• Country constraint: English FRS households only get assigned to English OAs, Welsh to Welsh, etc.
• Constituency collision avoidance: Each clone of the same household gets a different constituency (up to 50 retry iterations), ensuring geographic diversity across clones.

3. Tests — 19 passing, 1 skipped (NI)

Validates crosswalk completeness (OA counts, population totals, hierarchy nesting, country prefixes) and assignment correctness (country constraints, collision avoidance, population-weighted sampling, save/load roundtrip).

Known limitations

1. Northern Ireland excluded — NISRA download URLs returning 404. Handled gracefully.
2. Scotland OA population is uniform (~117 per OA) — NRS OA population CSV returning 403. Falls back to equal estimate.
3. No NI constituency mapping — needs separate lookup.

What comes next (Phases 2-6)

Phase 2: Clone-and-Assign

Clone each FRS household N times (start with N=10), assign each clone a different OA. Insert into `create_datasets.py` after imputations, before calibration.
US ref: PRs #457, #531

Phase 3: L0 Calibration Engine

Port L0-regularized optimization from US side. HardConcrete gates to actively drop records, producing sparse datasets. Add `l0-python` dependency.
US ref: PRs #364, #365

Phase 4: Sparse Matrix Builder

Build sparse `(n_targets × n_records*n_clones)` calibration matrix. Simulate PolicyEngine-UK per clone, wire existing `targets/sources/` into sparse matrix rows.
US ref: PRs #456, #489

Phase 5: SQLite Target Database

Hierarchical target storage: UK → Country → Region → LA → Constituency → MSOA → LSOA → OA. Migrate existing CSV/Excel targets into SQLite.
US ref: PRs #398, #488

Phase 6: Local Area Publishing

Generate per-area H5 files from sparse weights. Modal integration for scale.
US ref: PR #465

File summary

File	Lines	Purpose
`docs/oa_calibration_pipeline.md`	96	Full 6-phase roadmap
`calibration/init.py`	0	New calibration package
`calibration/oa_crosswalk.py`	878	Downloads & builds unified UK OA crosswalk from ONS/NRS/NISRA
`calibration/oa_assignment.py`	310	Population-weighted OA assignment with constraints
`storage/oa_crosswalk.csv.gz`	—	Pre-built crosswalk (235K areas, 1.4MB)
`tests/test_oa_crosswalk.py`	264	19 passing + 1 skipped test