feat: Enable DPP support with native_datafusion scan #3060
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
Query Plans (selectivity=0.01) |
Contributor
Author
Note: Spark: FileScan parquet (standard Spark scan) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3060 +/- ##
============================================
- Coverage 56.12% 54.59% -1.54%
- Complexity 976 1262 +286
============================================
Files 119 167 +48
Lines 11743 15522 +3779
Branches 2251 2574 +323
============================================
+ Hits 6591 8474 +1883
- Misses 4012 5830 +1818
- Partials 1140 1218 +78 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Shekharrajak
force-pushed
the
feature/dpp-phase1-mvp
branch
from
58f60e7 to
1b6f816
Compare
wForget
reviewed
Jan 10, 2026
|
|
||
| // Add runtime filter bounds if available | ||
| // These are pushed down from join operators to enable I/O reduction | ||
| addRuntimeFilterBounds(scan, nativeScanBuilder) |
Member
There was a problem hiding this comment.
The dataFilters have been pushed down to the native scan.
Contributor
Author
There was a problem hiding this comment.
thanks for sharing @wForget - looks like these changes were not required.
Contributor
Author
There was a problem hiding this comment.
Filters are pushed down but native filtering is not implemented, right ? I think those fallback to spark.
Member
There was a problem hiding this comment.
Filters are pushed down but native filtering is not implemented, right ? I think those fallback to spark.
There are several possibilities:
- Runtime Filters have been pushed down to the native parquet reader, but the parquet reader is not enabled or does not support them. (Parquet row group filters will be ineffective, but subsequent Filter operator will be effective.)
- Runtime Filters have been generated, but they caused the spark plan to fall back to vanilla spark. (As far as I know, Comet’s support for subqueries may not be complete.)
- Runtime Filters were not generated correctly.
Contributor
Author
There was a problem hiding this comment.
Thanks for sharing. I tried native runtime filters execution but do not see any improvements
The benchmark shows Comet is slower than Spark for workloads. This could be due to:
JNI Overhead: Significant cost crossing JVM/native boundary
Spark's Vectorized Reader: Highly optimized for in-memory parquet reading
Runtime Filters Not Effective: Filters may not be pruning row groups because:
Benchmark parquet files don't have bloom filters written
Data is in few row groups (small files)
Shekharrajak
force-pushed
the
feature/dpp-phase1-mvp
branch
from
2b5c120 to
1add4cf
Compare
Shekharrajak
commented
Jan 10, 2026
| withInfo(scanExec, s"Full native scan disabled because ${COMET_EXEC_ENABLED.key} disabled") | ||
| } | ||
|
|
||
| // Native DataFusion doesn't support subqueries/dynamic pruning |
Contributor
Author
There was a problem hiding this comment.
With or without this if condition - I am seeing same benchmarking, not sure when this is used .
Contributor
Author
|
Update: Although I could not achieve, what I was trying to do but the takeaway is : Even for non-partitioned tables where DPP operates via dataFilters (not partitionFilters), Comet's native DataFusion scan shows: Massive I/O reduction through efficient filter pushdown |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ref #3053
Rationale for this change
Dynamic Partition Pruning (DPP) was previously disabled for native_datafusion scan due to subquery handling concerns. However, Spark's DPP mechanism already evaluates dynamic filters and provides pre-filtered partition lists via dynamicallySelectedPartitions. This PR enables DPP support, achieving up to 301x I/O reduction by leveraging Spark's existing DPP infrastructure with native scan.
What changes are included in this PR?
How are these changes tested?
Unit tests
Add CometDPPSuite - test suite validating DPP with native scan
Add CometDPPBenchmark - benchmark showing I/O reduction metrics
Run the benchmark using