[GLUTEN-11921] Enable Parquet read/write test for NullType

[wecharyu]

What changes are proposed in this pull request?

Enable parquet NullType read/write test.

With excludeByPrefix("SPARK-54220") removed, "SPARK-54220: NullType" test is put back for testing. And "SPARK-54220: vectorized reader: missing all struct fields, struct with NullType only" test is replaced by a variant with a check removed since it is only applicable to vanilla Spark.

This PR depends on facebookincubator/velox#17391.

How was this patch tested?

Pass UT in GlutenParquetIOSuite.

Was this patch authored or co-authored using generative AI tooling?

No.

Related issue: #11921

[philo-he]

Thanks for the PR.

[philo-he]

Can we keep this test excluded in VeloxTestSettings.scala? That way, it won't affect other backends' test enabling controls.

[philo-he]

testGluten("SPARK-54220: xxx will add a prefix into the test name. I assume excludeByPrefix("SPARK-54220") will not exclude this test variant.

[wecharyu]

I'd prefer exclude it here because this test failure is not Velox-specific, it's caused by Gluten instead of backends.

Spark test collect leaf scan node FileSourceScanExecTransformer and run executeColumnar() in VerifyNoAdditionalScanOutputExec:

gluten/shims/spark41/src/main/scala/org/apache/spark/sql/execution/AbstractFileSourceScanExec.scala

Lines 139 to 142 in 356fda0

	override protected def doExecuteColumnar(): RDD[ColumnarBatch] = {
	val numOutputRows = longMetric("numOutputRows")
	val scanTime = longMetric("scanTime")
	inputRDD.asInstanceOf[RDD[ColumnarBatch]].mapPartitionsInternal {

The inputRDD is RDD[UnsafeRow] for parquetVectorizedReaderEnabled=false test path, so it would always failed by:

java.lang.ClassCastException: class org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to class org.apache.spark.sql.vectorized.ColumnarBatch

[philo-he]

Thanks for the clarification. I updated the PR description according to my understanding.

Please rebase the code and check if the CI reported failures are related. Thank you!

[wecharyu]

The failure is expected, we need first support it in Velox, PR is facebookincubator/velox#17391

[philo-he]

The failure is expected, we need first support it in Velox, PR is facebookincubator/velox#17391

@wecharyu, thank you for the clarification. If you need to validate the Velox PR, you can push a temporary commit with the PR ID added in the code below.

gluten/ep/build-velox/src/get-velox.sh

Line 28 in 97632d8

UPSTREAM_VELOX_PR_ID=""