Skip to content

Spark: Preserve SPJ regression tests from Spark upgrade verification drafts - #18995

Open
shivadarshan-devadiga wants to merge 1 commit into
apache:mainfrom
shivadarshan-devadiga:spark-preserve-spj-regression-tests
Open

shivadarshan-devadiga wants to merge 1 commit into
apache:mainfrom
shivadarshan-devadiga:spark-preserve-spj-regression-tests

Conversation

@shivadarshan-devadiga

Copy link
Copy Markdown

Closes #18948.

This keeps the TestStoragePartitionedJoins tests that @manuzhang wrote in the Spark verification drafts #18340 (Spark 4.0.5), #18339 (Spark 4.1.4) and #18294 (Spark 4.2.1) as permanent tests. The tests are taken from the drafts unchanged. In spark/v4.2 the new tests are placed before testJoinsHourToDays rather than after it, to avoid conflicting with #18990. #18383 also adds a test in that area, so whichever of #18383 and this PR merges second needs a rebase; I'm happy to do it if this one is second.

Coverage added:

Coverage 4.0 4.1 4.2
Broadcast joins ordered by a computed join-key expression, ascending and descending (SPARK-59905) ✓ ✓ ✓
Mixed-type partition keys with a non-leading subset join key, with partial clustering off and on (SPARK-60203); compatibility coverage on 4.2 ✓ ✓ ✓
Runtime filtering with a trailing partition key ✓ ✓
One-side shuffle with binary partition keys ✓ ✓
Chained or successive joins whose intermediate output has unknown partition keys ✓ ✓
Duplicate partition-key aliases, and different projected key subsets ✓
Columnar UNION ALL preserving key grouping ✓
Joins over checkpointed Iceberg scans, one or both sides checkpointed ✓

The aggregate over a partially clustered join is left to #18383, as the issue asks.

Testing:

  • The full TestStoragePartitionedJoins suite passes on the versions main pins: 80 tests on Spark 4.0.5, 84 on 4.1.4 and 72 on 4.2.1, with 0 failures.
  • On the releases before the fixes:
    • The broadcast-ordering tests fail on Spark 4.0.4, 4.1.3 and 4.2.0, with the rows in the wrong order (SPARK-59905).
    • subsetJoinKeysWithMixedPartitionTypes fails on 4.0.4 and 4.1.3 with IllegalStateException: Unknown type for int field. Type name: java.lang.String (SPARK-60203), and passes on 4.2.0, which was never affected.
    • partiallyClusteredSubsetJoinKeysWithMixedPartitionTypes passes on all three, so it is compatibility coverage.
  • spotlessCheck passes for all three modules.

…drafts

Co-authored-by: Manu Zhang <owenzhang1990@gmail.com>
@shivadarshan-devadiga

Copy link
Copy Markdown
Author

@manuzhang @szehon-ho Can you please take a look whenever you get a moment. Thanks!

@github-actions github-actions Bot added the spark label Nov 16, 2026
@shivadarshan-devadiga

Copy link
Copy Markdown
Author

Gentle ping @manuzhang @szehon-ho

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spark: Preserve SPJ regression tests from Spark upgrade verification drafts

1 participant