feat(rust/sedona-spatial-join): Support spatial partitioned spatial join for non-knn predicates

[Kontinuation]

This patch adds a probe-side partitioned stream provider for repartitioning the probe side, and create probe streams for specified partitions. This partitioned stream provider is integrated into the spatial join execution flow to support larger-than-memory dataset by breaking the data into smaller partitions, where each partition could be fully loaded into memory.

Currently only non-knn joins are supported. We'll add larger-than-memory KNN join support in subsequent patches.

[Copilot]

Pull request overview

This PR implements spatial partitioned stream processing for non-KNN spatial joins, enabling joins to handle larger-than-memory datasets by partitioning data into smaller chunks that can be loaded into memory. The main additions include a partitioned probe stream provider, updated stream state management for processing multiple partitions sequentially, and bitmap tracking for deduplicating probe rows across partitions in outer joins.

Changes:

• Added probe-side partitioned stream infrastructure to repartition and spill probe data
• Extended spatial join stream state machine to iterate through multiple spatial partitions
• Implemented bitmap tracking for probe-side Multi partition to handle right outer joins correctly

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
rust/sedona-spatial-join/src/utils/join_utils.rs	Added functions for tracking visited probe rows across partitions and adjusting indices with bitmap information
rust/sedona-spatial-join/src/stream.rs	Extended state machine to process multiple spatial partitions sequentially with proper bitmap tracking
rust/sedona-spatial-join/src/probe/partitioned_stream_provider.rs	New probe stream provider for repartitioning and spilling probe data to disk
rust/sedona-spatial-join/src/probe/non_partitioned_stream.rs	Wrapper stream for non-partitioned probe processing with metrics
rust/sedona-spatial-join/src/probe/first_pass_stream.rs	Stream implementation for first-pass processing that splits and spills probe data
rust/sedona-spatial-join/src/probe.rs	Module definition with shared probe metrics structure
rust/sedona-spatial-join/src/prepare.rs	Refactored to create probe stream options and spatial partitioners for probe side
rust/sedona-spatial-join/src/operand_evaluator.rs	Minor style improvement in conditional logic
rust/sedona-spatial-join/src/lib.rs	Added probe module to library exports
rust/sedona-spatial-join/src/index/partitioned_index_provider.rs	Exposed wait_for_index method for public use
rust/sedona-spatial-join/src/exec.rs	Updated to pass session config and runtime environment to stream constructor

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.