Skip to content

test: fix flaky mixed field id directory test on Spark 3.4 and 3.5 - #6312

Merged
andygrove merged 1 commit into
apache:mainfrom
andygrove:investigate-issue-6301
Sep 28, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
andygrove:investigate-issue-6301

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #6301.

Rationale for this change

The nightly failed on Spark 3.4 in ParquetReadV1Suite "a file without ids next to a file with ids is checked on its own", which #6116 added. The assertion that failed was on the Spark reference error, not Comet's, and the same test passed on Spark 3.4 in the previous nightly.

The test writes each side with sparkContext.parallelize on the five-core test session, so each write leaves an empty part-00000 next to two one-row files. The read packs the six files into three tasks: task 0 [ids, ids], task 1 [no ids, no ids] and task 2 [empty with ids, empty without ids]. On Spark 3.x, tasks 1 and 2 fail with different exceptions:

  • Task 1 fails on its first file and raises the bare RuntimeException from ParquetReadSupport.
  • Task 2 reads the file without ids after an empty file. Its error is raised inside the try { hasNext } in FileScanRDD.nextIterator, which wraps it in SparkException("Encountered error while reading file ...").

On 3.x the DAGScheduler wraps whichever task fails first in "Job aborted", and isMissingFieldIdsError looks only at getCause. The test passes when task 1 fails first and fails when task 2 does. The nightly log reports Task 2 in stage 2468.0 failed. Spark 3.5 has the same exposure. Spark 4.x wraps every read error once in FAILED_READ_FILE and does not add "Job aborted", so the test always passes there. Comet raises the same exception from every task on every version.

What changes are included in this PR?

The test writes each side with .repartition(1), which is how the suite already writes a single file. With one file per side there is no empty file, only one task fails, and Spark raises the bare RuntimeException every time. The assertion keeps Spark's own getCause form from ParquetFieldIdIOSuite. The test comment explains why each side is one file.

branch-1.1 has the same test through #6266 and will need the same change.

How are these changes tested?

The change is to the test itself. The race cannot be forced from the suite, so the fix was checked in two ways:

  • A plain Spark program (no Comet) repeats the test's writes and drains each read partition on its own. On Spark 3.4.3 and 3.5.9 it shows the three-task layout and the two exception shapes above. With .repartition(1) it shows two single-file tasks, and the one that fails raises the bare RuntimeException in 30 of 30 runs on each version.
  • The ParquetReadV1Suite tests with "ids" in their name (7 tests, including this one) pass locally on the default profile (Spark 4.1) and with -Pspark-3.4. On Spark 3.4 the test now writes two files, of 454 and 476 bytes: the same two-row files as the plain Spark program, and no empty file.

The run-all-spark-profiles label is on this PR, so the Comet suites run against Spark 3.4, 3.5, 4.0 and 4.2 before it lands.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request run-all-spark-profiles Run the Comet test suites against every Spark profile on this pull request, ahead of the merge queue test Testing related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nightly CI failed on 2026-09-28

2 participants