feat(rust/sedona-geoparquet): Support `geometry_columns` option in `read_parquet(..)` to mark additional geometry columns #560

2010YOUY01 · 2026-01-29T11:42:12Z

Closes #530

Motivation

Today, converting legacy Parquet files that store geometry as raw WKB payloads inside BINARY columns into GeoParquet requires a full SQL rewrite pipeline. Users must explicitly parse WKB, assign CRS, and reconstruct the geometry column before writing:

# geo_legacy.parquet schema
# - geo_bin: Binary (payload is WKB)
# - c1: Int32
# - c2: Int32

df = sd.read_parquet("/data/geo_legacy.parquet")

df = df.to_view("t", overwrite=True)

df = sd.sql("""
  SELECT
    ST_SetSRID(ST_GeomFromWKB(geo_bin), 4326) AS geometry,
    * EXCLUDE (geo_bin)
  FROM t
""")

df.to_parquet("geo_geoparquet.parquet")

This works, but it would be have a easier to use python API:

“Treat this binary column as a geometry column with encoding=WKB and CRS=EPSG:4326.”

This PR introduces a geometry_columns option on read_parquet() so legacy Parquet files can be interpreted as GeoParquet directly, without SQL rewriting.

Proposed Python API

Demo

df = sd.read_parquet(
    "/data/geo_legacy.parquet",
    geometry_columns={
        "geo_bin": {
            "encoding": "WKB",
            "crs": 4326,
        }
    },
)

df.to_parquet("geo_geoparquet.parquet")

Specification

            geometry_columns: Optional mapping of column name to GeoParquet column
                metadata (e.g., {"geom": {"encoding": "WKB"}}). Use this to mark
                binary WKB columns as geometry columns. Supported keys:
                - encoding: "WKB" (required)
                - crs: string (e.g., "EPSG:4326") or integer SRID (e.g., 4326).
                  If not provided, the default CRS is OGC:CRS84
                  (https://www.opengis.net/def/crs/OGC/1.3/CRS84), which means
                  the data in this column must be stored in longitude/latitude
                  based on the WGS84 datum.
                - edges: "planar" (default) or "spherical"
                Useful for:
                - Legacy Parquet files with Binary columns containing WKB payloads.
                - Overriding GeoParquet metadata when fields like `crs` are missing.
                Precedence:
                - If a column appears in both GeoParquet metadata and this option,
                  the geometry_columns entry takes precedence.
                Example:
                - For `geo.parquet(geo1: geometry, geo2: geometry, geo3: binary)`,
                  `read_parquet("geo.parquet", geometry_columns={"geo2": {...}, "geo3": ...})`
                  will override `geo2` metadata and treat `geo3` as a geometry column.
                Safety:
                - Columns specified here are not validated for WKB correctness.
                  Invalid WKB payloads may cause undefined behavior.

The key points:

The geometry columns specified in the option overrides what's already in the metadata, I think this can be useful if the metadata is missing some configurations like crs, and we can use this API to provide more details
No validation for now, this can be done in a follow-on PR

Key Changes

Parse python option fields into rust GeoParquetColumnMetadata struct
In the schema inference step, first infer the schema from GeoParquet metadata as before, next look at the options, to add/override additional geometry columns

2010YOUY01 · 2026-01-29T11:45:06Z

I would love to hear feedbacks on the design and specification(in PR writeup)

Once we reach agreement on that, I will:

Add comprehensive tests
Polish the implementation — so far I have only validated the high-level structure; I haven’t yet reviewed the details (e.g. metadata parsing) carefully

paleolimbot

Thank you for this!

I took a look at the whole thing but I know you're still working so feel free to ignore comments that aren't in scope.

Mostly I think you can avoid exposing GeoParquetMetadata via the options and just accept a string for that parameter. I believe serde_json can automatically deserialize that for you to avoid the parsing code here.

sedona-db/rust/sedona-geoparquet/src/metadata.rs

Lines 292 to 293 in 3f91e26

    
               /// Metadata about geometry columns. Each key is the name of a geometry column in the table. 
        
               pub columns: HashMap<String, GeoParquetColumnMetadata>,

Exposing a HashMap<GeoParquetColumnMetadata> in the options is OK, too if you feel strongly about it (probably helpful if this is being used from Rust), but for our bulit-in frontends (Python, R, SQL) a String is easier to deal with.

paleolimbot · 2026-01-29T15:10:18Z

python/sedonadb/python/sedonadb/context.py

        self,
        table_paths: Union[str, Path, Iterable[str]],
        options: Optional[Dict[str, Any]] = None,
+        geometry_columns: Optional[Dict[str, Union[str, Dict[str, Any]]]] = None,


I would probably make this just Optional[Mapping[str, Any]]. In Python a user can rather easily decode that from JSON if they need to.

paleolimbot · 2026-01-29T15:17:42Z

python/sedonadb/python/sedonadb/context.py

+                - encoding: "WKB" (required)
+                - crs: string (e.g., "EPSG:4326") or integer SRID (e.g., 4326).
+                  If not provided, the default CRS is OGC:CRS84
+                  (https://www.opengis.net/def/crs/OGC/1.3/CRS84), which means
+                  the data in this column must be stored in longitude/latitude
+                  based on the WGS84 datum.
+                - edges: "planar" (default) or "spherical"


Perhaps just "See the GeoParquet specification for the required fields for each column".

https://geoparquet.org/releases/v1.1.0/

paleolimbot · 2026-01-29T15:19:47Z

python/sedonadb/python/sedonadb/context.py

+                metadata (e.g., {"geom": {"encoding": "WKB"}}). Use this to mark
+                binary WKB columns as geometry columns. Supported keys:


Suggested change

metadata (e.g., {"geom": {"encoding": "WKB"}}). Use this to mark

binary WKB columns as geometry columns. Supported keys:

metadata (e.g., {"geom": {"encoding": "WKB"}}). Use this to mark

binary WKB columns as geometry columns or correct metadata such

as the column CRS. Supported keys:

paleolimbot · 2026-01-29T15:20:59Z

python/sedonadb/src/context.rs

+fn parse_geometry_columns<'py>(
+    py: Python<'py>,
+    geometry_columns: HashMap<String, PyObject>,
+) -> Result<HashMap<String, GeoParquetColumnMetadata>, PySedonaError> {


I think this bit can be avoided by just passing a string at this point (i.e., in Python, use json.dumps() before passing to Rust).

paleolimbot · 2026-01-29T15:21:55Z

python/sedonadb/src/context.rs

        py: Python<'py>,
        table_paths: Vec<String>,
        options: HashMap<String, PyObject>,
+        geometry_columns: Option<HashMap<String, PyObject>>,


Suggested change

geometry_columns: Option<HashMap<String, PyObject>>,

geometry_columns: Option<String>,

..I think JSON is the right format for this particular step (reduces bindings code considerably!)

paleolimbot · 2026-01-29T15:22:30Z

python/sedonadb/tests/test_context.py

+    src = tmp_path / "plain.parquet"
+    pq.write_table(table, src)
+
+    # Check metadata: geoparquet meatadata should not be available


Suggested change

# Check metadata: geoparquet meatadata should not be available

# Check metadata: geoparquet metadata should not be available

paleolimbot · 2026-01-29T15:24:59Z

python/sedonadb/tests/test_context.py

+    out = tmp_path / "geo.parquet"
+    df.to_parquet(out)
+    metadata = pq.read_metadata(out).metadata
+    assert metadata is not None
+    geo = metadata.get(b"geo")
+    assert geo is not None
+    geo_metadata = json.loads(geo.decode("utf-8"))
+    print(json.dumps(geo_metadata, indent=2, sort_keys=True))
+    assert geo_metadata["columns"]["geom"]["crs"] == "EPSG:4326"


I think you can probably skip this bit of the test (verifying the geometry-ness and CRS of the input seems reasonable to me).

paleolimbot · 2026-01-29T15:26:27Z

rust/sedona-geoparquet/src/provider.rs

 pub struct GeoParquetReadOptions<'a> {
    inner: ParquetReadOptions<'a>,
    table_options: Option<HashMap<String, String>>,
+    geometry_columns: Option<HashMap<String, GeoParquetColumnMetadata>>,


Suggested change

geometry_columns: Option<HashMap<String, GeoParquetColumnMetadata>>,

geometry_columns: Option<String>,

...just keeping this as a String will make this work for SQL, too (easy to import from the HashMap<String, String> that DataFusion gives us)

paleolimbot · 2026-01-29T15:27:42Z

rust/sedona-schema/src/crs.rs


    // Handle JSON strings "OGC:CRS84", "EPSG:4326", "{AUTH}:{CODE}" and "0"
-    let crs = if LngLat::is_str_lnglat(crs_str) {
+    let crs = if crs_str == "OGC:CRS84" {


These changes should be reverted (there is >1 string that can represent lon/lat)

paleolimbot · 2026-01-29T15:28:23Z

rust/sedona-schema/src/crs.rs

        }
    }

+    if let Some(number) = crs_value.as_number() {


This part is OK (but perhaps add a test)

2010YOUY01 · 2026-01-30T07:09:41Z

Mostly I think you can avoid exposing GeoParquetMetadata via the options and just accept a string for that parameter. I believe serde_json can automatically deserialize that for you to avoid the parsing code here.

sedona-db/rust/sedona-geoparquet/src/metadata.rs

Lines 292 to 293 in 3f91e26

/// Metadata about geometry columns. Each key is the name of a geometry column in the table.

pub columns: HashMap<String, GeoParquetColumnMetadata>,

Exposing a HashMap<GeoParquetColumnMetadata> in the options is OK, too if you feel strongly about it (probably helpful if this is being used from Rust), but for our bulit-in frontends (Python, R, SQL) a String is easier to deal with.

It's a great idea to make the rust internal API easier to use for different frontend bindings, WDYT:

For the builder of Rust Struct GeoParquetReadOptions, it takes string for Json options to make it easier to use
Inside GeoParquetReadOptions, it keeps typed/parsed field for geometry_columns, to make the rust backend impl cleaner

Now the API looks like:

pub struct GeoParquetReadOptions<'a> {
    inner: ParquetReadOptions<'a>,
    table_options: Option<HashMap<String, String>>,
    // Keep it typed to make backend impl cleaner
    geometry_columns: Option<HashMap<String, GeoParquetColumnMetadata>>,
}

impl GeoParquetReadOptions {
    // ...
    pub fn with_geometry_columns(
        mut self,
        // Json config string like "{"geo": {"encoding": "wkb"}}"
        geometry_columns: String,
    ) -> Self {
        let parsed = parse_geometry_columns(geometry_columns);
        self.geometry_columns = Some(parsed);
        self
    }
}

paleolimbot · 2026-01-30T14:41:32Z

Thank you for considering...sounds good to me!

2010YOUY01 added 2 commits January 29, 2026 19:27

feat: support geometry_columns option in Python read_parquet() API

a2b4d12

Merge branch 'main' into read-parquet-opt

fe78e42

2010YOUY01 marked this pull request as draft January 29, 2026 11:49

paleolimbot reviewed Jan 29, 2026

View reviewed changes

	/// Metadata about geometry columns. Each key is the name of a geometry column in the table.
	pub columns: HashMap<String, GeoParquetColumnMetadata>,

		metadata (e.g., {"geom": {"encoding": "WKB"}}). Use this to mark
		binary WKB columns as geometry columns. Supported keys:

	geometry_columns: Option<HashMap<String, PyObject>>,
	geometry_columns: Option<String>,

	# Check metadata: geoparquet meatadata should not be available
	# Check metadata: geoparquet metadata should not be available

feat(rust/sedona-geoparquet): Support geometry_columns option in read_parquet(..) to mark additional geometry columns #560

Are you sure you want to change the base?

feat(rust/sedona-geoparquet): Support geometry_columns option in read_parquet(..) to mark additional geometry columns #560

Uh oh!

Conversation

2010YOUY01 commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Proposed Python API

Demo

Specification

Key Changes

Uh oh!

2010YOUY01 commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paleolimbot commented Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(rust/sedona-geoparquet): Support `geometry_columns` option in `read_parquet(..)` to mark additional geometry columns #560

feat(rust/sedona-geoparquet): Support `geometry_columns` option in `read_parquet(..)` to mark additional geometry columns #560

2010YOUY01 commented Jan 29, 2026 •

edited

Loading

2010YOUY01 commented Jan 29, 2026 •

edited

Loading

2010YOUY01 commented Jan 30, 2026 •

edited

Loading