Repository navigation
Spark: Preserve SPJ regression tests from Spark upgrade verification drafts - #18995
Open
shivadarshan-devadiga wants to merge 1 commit into
Open
shivadarshan-devadiga wants to merge 1 commit into
shivadarshan-devadiga wants to merge 1 commit into
Conversation
…drafts Co-authored-by: Manu Zhang <owenzhang1990@gmail.com>
Author
|
@manuzhang @szehon-ho Can you please take a look whenever you get a moment. Thanks! |
Author
|
Gentle ping @manuzhang @szehon-ho |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18948.
This keeps the
TestStoragePartitionedJoinstests that @manuzhang wrote in the Spark verification drafts #18340 (Spark 4.0.5), #18339 (Spark 4.1.4) and #18294 (Spark 4.2.1) as permanent tests. The tests are taken from the drafts unchanged. Inspark/v4.2the new tests are placed beforetestJoinsHourToDaysrather than after it, to avoid conflicting with #18990. #18383 also adds a test in that area, so whichever of #18383 and this PR merges second needs a rebase; I'm happy to do it if this one is second.Coverage added:
UNION ALLpreserving key groupingThe aggregate over a partially clustered join is left to #18383, as the issue asks.
Testing:
TestStoragePartitionedJoinssuite passes on the versions main pins: 80 tests on Spark 4.0.5, 84 on 4.1.4 and 72 on 4.2.1, with 0 failures.subsetJoinKeysWithMixedPartitionTypesfails on 4.0.4 and 4.1.3 withIllegalStateException: Unknown type for int field. Type name: java.lang.String(SPARK-60203), and passes on 4.2.0, which was never affected.partiallyClusteredSubsetJoinKeysWithMixedPartitionTypespasses on all three, so it is compatibility coverage.spotlessCheckpasses for all three modules.