chore: run Spark 3.4 tests with `native_datafusion` scan by andygrove · Pull Request #3722 · apache/datafusion-comet

andygrove · 2026-03-17T15:43:21Z

Which issue does this PR close?

Part of #3321

Related to #3311

Rationale for this change

The native_datafusion Spark SQL tests were already running for Spark 3.5 but not for Spark 3.4. Adding 3.4 coverage revealed test failures that need to be skipped with the appropriate ignore tags.

What changes are included in this PR?

Add Spark 3.4.3 to the native_datafusion CI workflow
Update dev/diffs/3.4.3.diff to tag failing tests with IgnoreCometNativeDataFusion and IgnoreCometNativeScan, matching tags already applied in the 3.5.8 diff plus 3 tests specific to Spark 3.4:
- FileBasedDataSourceSuite - "Spark native readers should respect spark.sql.caseSensitive"
- ParquetIOSuite - "SPARK-35640: read binary as timestamp should throw schema incompatible error"
- ParquetIOSuite - "SPARK-35640: int as long should throw schema incompatible error"
- ParquetQuerySuite - "SPARK-36182: can't read TimestampLTZ as TimestampNTZ"
- ParquetQuerySuite - "SPARK-34212 Parquet should read decimals correctly"
- ParquetQuerySuite - "row group skipping doesn't overflow when reading into larger type"
- ParquetSchemaSuite - "schema mismatch failure error message for parquet vectorized reader"
- ParquetSchemaSuite - "SPARK-45604: schema mismatch failure error on timestamp_ntz to array<timestamp_ntz>"
- ParquetFilterSuite - "filter pushdown - StringPredicate" (IgnoreCometNativeScan)
- ParquetFilterSuite - "SPARK-25207: exception when duplicate fields in case-insensitive mode"
- DynamicPartitionPruningSuite - "static scan metrics"
- ExtractPythonUDFsSuite - "Python UDF should not break column pruning/filter pushdown -- Parquet V1"

How are these changes tested?

By running the Spark SQL native_datafusion tests in CI for Spark 3.4.3.

Tag tests with IgnoreCometNativeDataFusion and IgnoreCometNativeScan to match tags already applied in the 3.5.8 diff, plus 3 tests that are specific to Spark 3.4.

…diff Remove imports of IgnoreCometNativeDataFusion in DynamicPartitionPruningSuite and FileBasedDataSourceSuite since both files are in the org.apache.spark.sql package where IgnoreCometNativeDataFusion is defined, making the import redundant and causing "permanently hidden" compilation errors.

…fusion-3.4

… diff The test override in SQLTestUtils checked DisableAdaptiveExecution first, so tests with both DisableAdaptiveExecution and IgnoreCometNativeDataFusion tags (like "static scan metrics") would enter the DAE branch and never check the ignore tag. Reorder to match the 3.5.8 diff: check Comet skip tags first with early returns, then handle DisableAdaptiveExecution.

parthchandra · 2026-03-20T16:44:16Z

dev/diffs/3.4.3.diff

             assert(bJoinExec.isEmpty)
             val smJoinExec = collect(joinedDF.queryExecution.executedPlan) {
               case smJoin: SortMergeJoinExec => smJoin
 +              case smJoin: CometSortMergeJoinExec => smJoin


what's the change here (and in the two changed lines below)?

There is no change here. The + at the start is not indicating that anything changed in this PR. This is part of the diff that is already on main.

parthchandra · 2026-03-20T17:09:26Z

dev/diffs/3.4.3.diff


           val scanNodes = query.queryExecution.executedPlan.collect {
             case scan: FileSourceScanExec => scan
 +            case scan: CometScanExec => scan


Is this not sufficient to enable the test?

No. We would need to add CometNativeScanExec here to support native_datafusion. I am not sure if that is sufficient to actually make the test pass though.

Again, this code is not changed in the PR - the test is just ignored in this PR for native_datafusion scan.

comphead · 2026-03-20T17:47:28Z

I'll check it this afternoon

kazuyukitanimura · 2026-03-20T18:34:08Z

dev/diffs/3.4.3.diff

+   }
+
+-  test("filter pushdown - StringPredicate") {
+  test("filter pushdown - StringPredicate", IgnoreCometNativeScan("cannot be pushed down")) {


Is this a new issue?

hmm this should have been ignored with IgnoreCometDataFusionScan not IgnoreCometNativeScan. I will update this.

do we have existing GH issue link to follow this later?

I went ahead and updated this in this PR

comphead

Thanks @andygrove looks like #3311 is a biggest villain for this PR, marking the issue so we can reenable spark tests once #3311 fixed

….4.3 diff

… diff - static scan metrics → apache#3442 (DPP not supported in native_datafusion) - caseSensitive and SPARK-25207 duplicate fields → apache#3760 (new issue) - Python UDF filter pushdown → fixed by adding CometNativeScanExec pattern matches, removed IgnoreCometNativeDataFusion tag

andygrove · 2026-03-22T17:49:44Z

Thanks @andygrove looks like #3311 is a biggest villain for this PR, marking the issue so we can reenable spark tests once #3311 fixed

I updated the diff to link to new specific issues so that we can close the old #3311 issue which covered multiple ~~bugs~~ incompatibilities

…nUDFsSuite

andygrove added 5 commits March 17, 2026 09:42

run Spark 3.4 tests with native_datafusion scan

93ae842

fix: update 3.4.3 diff to ignore failing native_datafusion tests

7fd9655

Tag tests with IgnoreCometNativeDataFusion and IgnoreCometNativeScan to match tags already applied in the 3.5.8 diff, plus 3 tests that are specific to Spark 3.4.

Merge remote-tracking branch 'apache/main' into spark-sql-native-data…

60368c7

…fusion-3.4

andygrove changed the title ~~chore: run Spark 3.4 tests with native_datafusion scan [WIP]~~ chore: run Spark 3.4 tests with native_datafusion scan Mar 18, 2026

andygrove marked this pull request as ready for review March 18, 2026 18:40

andygrove requested review from comphead and parthchandra March 20, 2026 14:08

parthchandra reviewed Mar 20, 2026

View reviewed changes

kazuyukitanimura reviewed Mar 20, 2026

View reviewed changes

comphead approved these changes Mar 22, 2026

View reviewed changes

andygrove added 4 commits March 22, 2026 08:21

address feedback

9292ff9

fix

2e92eb5

fix: retag schema mismatch tests from apache#3311 to apache#3720 in 3…

e9502bd

….4.3 diff

fix: remove unused IgnoreCometNativeDataFusion import in ExtractPytho…

6e3ce77

…nUDFsSuite

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: run Spark 3.4 tests with `native_datafusion` scan#3722

chore: run Spark 3.4 tests with `native_datafusion` scan#3722
andygrove wants to merge 10 commits intoapache:mainfrom
andygrove:spark-sql-native-datafusion-3.4

andygrove commented Mar 17, 2026 •

edited

Loading

Uh oh!

parthchandra Mar 20, 2026

Uh oh!

andygrove Mar 20, 2026

Uh oh!

parthchandra Mar 20, 2026

Uh oh!

andygrove Mar 20, 2026

Uh oh!

andygrove Mar 20, 2026

Uh oh!

comphead commented Mar 20, 2026

Uh oh!

kazuyukitanimura Mar 20, 2026

Uh oh!

andygrove Mar 20, 2026

Uh oh!

comphead Mar 22, 2026

Uh oh!

andygrove Mar 22, 2026

Uh oh!

comphead left a comment

Uh oh!

andygrove commented Mar 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

andygrove commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead commented Mar 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

andygrove commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andygrove commented Mar 17, 2026 •

edited

Loading

andygrove commented Mar 22, 2026 •

edited

Loading