Skip to content

Spark 4.2: Test INSERT into a computed IDENTIFIER target - #18986

Open
shivadarshan-devadiga wants to merge 1 commit into
apache:mainfrom
shivadarshan-devadiga:spark-test-insert-computed-identifier
Open

shivadarshan-devadiga wants to merge 1 commit into
apache:mainfrom
shivadarshan-devadiga:spark-test-insert-computed-identifier

Conversation

@shivadarshan-devadiga

Copy link
Copy Markdown

Closes #18950.

SPARK-58933 fixes expression resolution in the target of INSERT INTO IDENTIFIER(...). This adds testInsertIntoComputedIdentifier to TestSparkCatalogOperations in spark/v4.2: it inserts into IDENTIFIER(lower('<table>')) and checks the inserted row. The test is the insertIntoComputedIdentifier regression from the Spark 4.2.1 verification draft #18294 by @manuzhang, kept as a permanent test.

It is added to Spark 4.2 only. The bug is in Spark 4.2.0 and fixed in 4.2.1. On Spark 4.1.4, the version spark/v4.1 uses, the same test passes for all 4 catalogs.

Testing:

  • The new test passes for all 4 catalog configurations of the suite on Spark 4.2.1, the version on main. The rest of TestSparkCatalogOperations passes too.
  • With spark42 set to 4.2.0, which doesn't have SPARK-58933, all 4 runs fail with UNSUPPORTED_INSERT.RDD_BASED: Can't insert into the target.
  • Added temporarily to spark/v4.1 on Spark 4.1.4, it passes for all 4 catalogs.
  • spotlessCheck passes.

@github-actions github-actions Bot added the spark label Nov 16, 2026
@shivadarshan-devadiga

shivadarshan-devadiga commented Nov 16, 2026 •

Copy link
Copy Markdown
Author

@manuzhang @szehon-ho Can you please take a look whenever you get a moment. Thanks!

Co-authored-by: Manu Zhang <owenzhang1990@gmail.com>
@shivadarshan-devadiga
shivadarshan-devadiga force-pushed the spark-test-insert-computed-identifier branch from d83fa0d to d0f7008 Compare November 16, 2026 08:40
@shivadarshan-devadiga

Copy link
Copy Markdown
Author

Gentle ping @manuzhang @szehon-ho

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spark: Test INSERT targets with computed IDENTIFIER expressions

1 participant