Repository navigation
test: fix flaky mixed field id directory test on Spark 3.4 and 3.5 - #6312
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6301.
Rationale for this change
The nightly failed on Spark 3.4 in
ParquetReadV1Suite"a file without ids next to a file with ids is checked on its own", which #6116 added. The assertion that failed was on the Spark reference error, not Comet's, and the same test passed on Spark 3.4 in the previous nightly.The test writes each side with
sparkContext.parallelizeon the five-core test session, so each write leaves an emptypart-00000next to two one-row files. The read packs the six files into three tasks: task 0[ids, ids], task 1[no ids, no ids]and task 2[empty with ids, empty without ids]. On Spark 3.x, tasks 1 and 2 fail with different exceptions:RuntimeExceptionfromParquetReadSupport.try { hasNext }inFileScanRDD.nextIterator, which wraps it inSparkException("Encountered error while reading file ...").On 3.x the
DAGSchedulerwraps whichever task fails first in "Job aborted", andisMissingFieldIdsErrorlooks only atgetCause. The test passes when task 1 fails first and fails when task 2 does. The nightly log reportsTask 2 in stage 2468.0 failed. Spark 3.5 has the same exposure. Spark 4.x wraps every read error once inFAILED_READ_FILEand does not add "Job aborted", so the test always passes there. Comet raises the same exception from every task on every version.What changes are included in this PR?
The test writes each side with
.repartition(1), which is how the suite already writes a single file. With one file per side there is no empty file, only one task fails, and Spark raises the bareRuntimeExceptionevery time. The assertion keeps Spark's owngetCauseform fromParquetFieldIdIOSuite. The test comment explains why each side is one file.branch-1.1has the same test through #6266 and will need the same change.How are these changes tested?
The change is to the test itself. The race cannot be forced from the suite, so the fix was checked in two ways:
.repartition(1)it shows two single-file tasks, and the one that fails raises the bareRuntimeExceptionin 30 of 30 runs on each version.ParquetReadV1Suitetests with "ids" in their name (7 tests, including this one) pass locally on the default profile (Spark 4.1) and with-Pspark-3.4. On Spark 3.4 the test now writes two files, of 454 and 476 bytes: the same two-row files as the plain Spark program, and no empty file.The
run-all-spark-profileslabel is on this PR, so the Comet suites run against Spark 3.4, 3.5, 4.0 and 4.2 before it lands.