Repository navigation
Conversation
…e as Iceberg views The Hive catalog stores Iceberg views in the Hive Metastore as Hive views. When SparkSessionCatalog fails to load such a view as an Iceberg table, it falls back to the session catalog, which loads it as a Hive view. Spark then resolves it as a v1 view before the ResolveViews rule of the Iceberg extensions can, by name with the view's column names, which fails when the view has column aliases: [INCOMPATIBLE_VIEW_SCHEMA_CHANGE] ... column c1 cannot be resolved. Expected 1 columns named c1 but got []. Don't return a view of the session catalog that is an Iceberg view, so that it is resolved as an Iceberg view, as it is with the other catalogs. Closes apache#17288 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #17288
Problem
With
spark_catalogset toSparkSessionCatalogover a Hive catalog, reading an Iceberg view that has column aliases fails:WITH SCHEMA EVOLUTIONfrom the issue isn't required. The column aliases are what trigger it.The Hive catalog stores Iceberg views in the Hive Metastore as
VIRTUAL_VIEWs.SparkSessionCatalog.loadTablefails to load the view as an Iceberg table and falls back to the session catalog.V1Table).ResolveRelationsthen resolves it as a v1 view before theResolveViewsrule of the Iceberg extensions gets to it.SessionCatalog.fromCatalogTablelooks up each view column by name in the output of the view's query. Without theview.query.out.col.*properties that Spark writes for its own views, it looks for the view's column names (c1). The query only outputscol1andcol2, so the lookup fails.Iceberg views are also missed by the rest of the Iceberg view handling in this setup:
DESCRIBE,SHOW CREATE TABLE, the schema binding mode, temp object checks, and others.Fix
In
SparkSessionCatalog.loadTable, the session catalog's result is not returned when it is a view that the Iceberg view catalog knows (viewExists). The originalNoSuchTableExceptionis rethrown instead, soResolveViewsresolves the view as an Iceberg view, the same as for the other catalogs. The extraviewExistscall is only made when the session catalog returns a view. Hive views that aren't Iceberg views are still resolved by Spark, becauseHiveCatalog.viewExistsis false for them.Testing
New
TestSessionCatalogHiveViewscoversSparkSessionCatalogwithtype=hive, whichTestViewsdoesn't cover. It has three tests:VALUESWITH SCHEMA EVOLUTIONAll three fail without the fix with the error above.
Locally, I also ran
TestViewswith only theSparkSessionCatalog+ Hive configuration as its parameter. That run isn't part of this PR.renameViewToDifferentTargetCatalog, fails with or without the fix.HiveCatalog.renameViewrejects the three-part target identifier, which is a separate issue.On Spark 4.2,
TestSparkSessionCatalog,TestViews,TestViewCatalogCache,TestReferencedByViewChain,TestTempViewRefreshandTestSessionCatalogHiveViewspass.Spark 3.5, 4.0 and 4.1 have the same code. I'll backport in a follow-up once this is reviewed.