spark: pass format options into reader

[a10y]

Summary

In Spark, pass format properties into reader.

This mirrors the existing implementation of ParquetDataSourceV2 which rips the relevant S3/Azure/etc. properties off of the active SparkSession's hadoopConf. This is also similar to what we do in Vortex iceberg already.

Testing

Added a new test suite for reading/writing to S3 using the Adobe S3Mock testcontainer.

[codspeed-hq]

Merging this PR will degrade performance by 15.51%

❌ 3 regressed benchmarks
✅ 1103 untouched benchmarks
⏩ 1522 skipped benchmarks¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	bitwise_not_vortex_buffer_mut[1024]	477.2 ns	535.6 ns	-10.89%
❌	Simulation	bitwise_not_vortex_buffer_mut[128]	317.8 ns	376.1 ns	-15.51%
❌	Simulation	map_each[BufferMut<i32>, 128]	770.6 ns	858.1 ns	-10.2%

---

_{Comparing aduffy/spark-props (64c3336) with develop (8aae4ba)}

Footnotes

1. 1522 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[a10y]

this is the thing we needed. everything else is just plumbing

[a10y]

i ripped this straight out of our iceberg fork

[a10y]

@claude please review. make no mistakes. be extremely javapilled

[a10y]

@claude please re-review. I should have addressed all of the RED comments, and most of the green ones. I don't think the YELLOW ones are relevant.

as always, stay javapilled.

[a10y]

@claude ok i addressed the props in cache key thing. happy now?

[claude]

Claude finished @a10y's task in 30s —— View job

---

Re-Review (Cache Key Fix)

• Read vortex-jni/src/object_store.rs — check cache key fix
• Post results

---

✅ Yes, happy now.

url_cache_key now accepts properties and folds the sorted key-value pairs into the key:

fn url_cache_key(url: &Url, properties: &HashMap<String, String>) -> String {
    let mut sorted_props: Vec<_> = properties.iter().collect();
    sorted_props.sort_by_key(|(k, _)| *k);
    let props_str: String = sorted_props
        .iter()
        .map(|(k, v)| format!("{k}={v}"))
        .collect::<Vec<_>>()
        .join(",");
    format!("{}://{};{}", url.scheme(), &url[...], props_str)
}

The sort guarantees determinism regardless of insertion order, so two calls with the same credentials always resolve to the same cache entry, and two calls with different credentials (or no credentials vs. credentials) get distinct entries. The fix is correct.

One non-blocking observation: credentials end up as plaintext in an in-memory HashMap key. That's fine — it's never serialized or logged — but worth being aware of if this cache ever grows legs (e.g., persisted to disk, exposed via metrics). Not asking for a change, just flagging it.

All previously raised issues are now resolved. LGTM. 🚀