feat(datafusion): upgrade to DataFusion 53 and use VERSION AS OF#236
feat(datafusion): upgrade to DataFusion 53 and use VERSION AS OF#236JingsongLi wants to merge 1 commit intoapache:mainfrom
Conversation
…E AS OF with VERSION AS OF Upgrade DataFusion from 52.3 to 53.0 (arrow/parquet 57→58, sqlparser 0.59→0.61, orc-rust 0.7→0.8, pyo3 0.26→0.28) and replace the old `FOR SYSTEM_TIME AS OF` time travel syntax with the new `VERSION AS OF` and `TIMESTAMP AS OF` syntax supported by sqlparser 0.61. Introduce `scan.version` option to unify snapshot id and tag name resolution: at scan time, the version value is resolved by first checking if a tag with that name exists, then trying to parse it as a snapshot id, otherwise returning an error. Remove the now-redundant `scan.snapshot-id` and `scan.tag-name` options.
|
+1 |
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
Overall looks good. The DataFusion 53 upgrade is well-contained and the unified scan.version design is cleaner than the previous dual-option approach.
Minor notes (non-blocking):
-
scan.versiontag-first resolution: A tag named1would shadow snapshot ID1. Worth a short doc note as a known edge case, though practically unlikely since most tag names are descriptive. -
Error message in
resolve_to_snapshot: TheTag {v} doesn't exist.branch inside thetag_existstrue path is technically unreachable (since we already confirmed the tag exists). Could be tightened up but not a blocker. -
Python test skip: Correctly skipped since the Python
datafusionpackage has not shipped 53 yet.
+1, good to merge.
|
https://lists.apache.org/thread/nps36jh2frp8wfh71564vo2vfozyo7vh Wait datafusion-python to release 53. |
3 participants
Purpose
Upgrade DataFusion from 52.3 to 53.0 (arrow/parquet 57→58, sqlparser 0.59→0.61, orc-rust 0.7→0.8, pyo3 0.26→0.28) and replace the old
FOR SYSTEM_TIME AS OFtime travel syntax with the newVERSION AS OFandTIMESTAMP AS OFsyntax supported by sqlparser 0.61.Introduce
scan.versionoption to unify snapshot id and tag name resolution: at scan time, the version value is resolved by first checking if a tag with that name exists, then trying to parse it as a snapshot id, otherwise returning an error. Remove the now-redundantscan.snapshot-idandscan.tag-nameoptions.Brief change log
Tests
API and Format
Documentation