[CORE] Fix Delta 4.0 Spark 4.1 package build#12078
Conversation
malinjawi
force-pushed
the
fix/delta40-nightly-package
branch
from
0560d84 to
f8e7c1f
Compare
malinjawi
changed the title
| run: | | ||
| cd $GITHUB_WORKSPACE/ | ||
| $MVN_CMD clean test-compile -Pspark-4.0 -Pscala-2.13 -Pbackends-velox -Pspark-ut -Piceberg,iceberg-test,delta,paimon -Pfast-build | ||
| $MVN_CMD clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta -Pfast-build |
There was a problem hiding this comment.
the fast-fail test is designed for AI tool
could we enable -Pdelta for Spark-4.1 test?
https://github.com/apache/gluten/blob/main/.github/workflows/velox_backend_x86.yml#L1378
There was a problem hiding this comment.
Thanks @zhouyuan. Got it, I moved the coverage from the fast-build job to the regular Spark 4.1 Velox test command, so spark-test-spark41 now runs with -Pdelta.
malinjawi
force-pushed
the
fix/delta40-nightly-package
branch
from
f8e7c1f to
8cb01ca
Compare
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.spark.sql.execution.streaming |
There was a problem hiding this comment.
it seems this class is introduced in this patch, is this a must?
There was a problem hiding this comment.
Rechecked this and agreed it should not live in the Spark 4.1 shim jar. I moved the compatibility wrapper into backends-velox/src-delta40-spark41/test, wired that test source root only under the Spark 4.1 profile, and deleted the production shim source. Verified locally with ./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta -DskipTests -Dmaven.source.skip.
| export PATH=$JAVA_HOME/bin:$PATH | ||
| java -version | ||
| $MVN_CMD clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox \ | ||
| -Pspark-ut -Pdelta -DskipTests -Dmaven.source.skip |
There was a problem hiding this comment.
instead of add this line, please add the -Pdelta in below line
…local # Conflicts: # .github/workflows/velox_backend_x86.yml
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
2 participants
What changes were proposed in this pull request?
This PR fixes the nightly package failure for the Spark 4.1 + Delta Velox build path.
Changes:
Loggingplus explicitSparkMDC.of(...)calls in Delta 4.0 logging code so the code compiles across Spark 4.0, Spark 4.1, Delta 4.0, and Delta 4.1.CheckpointFileManagerandMemoryStreampackages.delta-spark_4.1_2.13:4.1.0, from the Spark 4.1 Maven profile. Delta 4.0.0 still references SparkTimeAdd, which is absent in Spark 4.1.-Pdeltatest-compile coverage inside the regularspark-test-spark41workflow before running the existing Spark 4.1 unit tests.Why are the changes needed?
The nightly package build compiles Spark 4.1 with
-Pdelta, but PR CI did not enable Delta on the Spark 4.1 Velox path. Spark 4.1 changed the logging MDC shape and moved streaming APIs used by Delta 4.0, so the original issue only surfaced in nightly packaging.After enabling Spark 4.1 Delta coverage in PR CI, two more Spark 4.1 issues surfaced: Delta 4.0.0 references
org.apache.spark.sql.catalyst.expressions.TimeAdd, which is not present in Spark 4.1, and the reused Delta 4.0 tests need Spark 4.1-specific error expectations. The Spark 4.1 profile now resolves Delta 4.1.0 using the Spark-specific Delta artifact, and the tests select the Spark 4.1 expectations usingorg.apache.spark.SPARK_VERSION.How was this patch tested?
./build/mvn -ntp -pl shims/spark41 clean install -Pspark-4.1 -Pscala-2.13 -Pjava-21 -DskipTests -Dmaven.source.skip./build/mvn -ntp -pl backends-velox -am clean install -Pspark-4.1 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Pdelta -DskipTests -Dmaven.source.skip./build/mvn -ntp -pl backends-velox -am clean install -Pspark-4.0 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Piceberg -Phudi -Pdelta -Ppaimon -DskipTests -Dmaven.source.skip./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Pdelta -Ddelta.package.name=delta-spark_4.1 -Ddelta.version=4.1.0 -Ddelta.binary.version=40 -DskipTests -Dmaven.source.skip./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Pdelta -DskipTests -Dmaven.source.skip./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.0 -Pscala-2.13 -Pjava-21 -Pbackends-velox -Pceleborn -Puniffle -Pdelta -DskipTests -Dmaven.source.skip./build/mvn -ntp -pl backends-velox -am clean test-compile -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta -DskipTests -Dmaven.source.skip./build/mvn -ntp -q help:evaluate -Dexpression=delta.package.name -DforceStdout -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta./build/mvn -ntp -q help:evaluate -Dexpression=delta.version -DforceStdout -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdeltazipgrep -n TimeAdd ~/.m2/repository/io/delta/delta-spark_4.1_2.13/4.1.0/delta-spark_4.1_2.13-4.1.0.jargit diff --check