Fix Google Drive spreadsheet crawler table grouping

[Tian-2017]

Why

Google Drive spreadsheet imports write raw data under a dataset-specific path:

<department>/<output_folder>/<data_set_name>/import_year=...

The crawler was scanning the shared <department>/<output_folder> prefix with table level 3. For some imports, Glue grouped the dataset folder as a partition and created a generic table such as g_drive instead of the expected dataset table.

This caused land_reg_registered_leases_greater_london_2026_04 to land successfully in S3 but not appear as its own raw-zone Glue table.

Parking imports appeared to work because the parking g-drive folder contains many datasets with different schemas. In that case, Glue could not safely group everything into one shared g_drive table, so it created dataset-specific tables such as hackney_carpark with a warning message "Found schemas don't match at level 3; Created table hackney_carpark". That behaviour is incidental, not guaranteed. The crawler config is still too broad and should be fixed so every Google Drive import reliably crawls its own dataset path.

What Changed

• Updated the spreadsheet import crawler target to scan the dataset-specific raw prefix.
• Updated the crawler table level to match that path so the dataset folder becomes the table root.
• Keeps the Glue job output path unchanged.

Validation

• Manually tested the equivalent crawler config in prod.
• Confirmed Glue created housing-raw-zone.land_reg_registered_leases_greater_london_2026_04.
• Confirmed the table has the expected import_year, import_month, import_day, and import_date partitions.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud