Skip to content

Spark 4.2: Resolve Iceberg views in SparkSessionCatalog backed by Hive as Iceberg views - #18989

Open
raviston wants to merge 1 commit into
apache:mainfrom
raviston:spark-session-hive-view-aliases
Open

raviston wants to merge 1 commit into
apache:mainfrom
raviston:spark-session-hive-view-aliases

Conversation

@raviston

@raviston raviston commented Nov 16, 2026 •

Copy link
Copy Markdown

Closes #17288

Problem

With spark_catalog set to SparkSessionCatalog over a Hive catalog, reading an Iceberg view that has column aliases fails:

CREATE VIEW v (c1, c2) AS VALUES (1, 2);
SELECT * FROM v;
-- [INCOMPATIBLE_VIEW_SCHEMA_CHANGE] The SQL query of view `spark_catalog`.`default`.`v` has an incompatible
-- schema change and column c1 cannot be resolved. Expected 1 columns named c1 but got [].

WITH SCHEMA EVOLUTION from the issue isn't required. The column aliases are what trigger it.

The Hive catalog stores Iceberg views in the Hive Metastore as VIRTUAL_VIEWs.

  1. SparkSessionCatalog.loadTable fails to load the view as an Iceberg table and falls back to the session catalog.
  2. The session catalog returns the metastore entry as a Hive view (V1Table).
  3. Spark's ResolveRelations then resolves it as a v1 view before the ResolveViews rule of the Iceberg extensions gets to it.
  4. SessionCatalog.fromCatalogTable looks up each view column by name in the output of the view's query. Without the view.query.out.col.* properties that Spark writes for its own views, it looks for the view's column names (c1). The query only outputs col1 and col2, so the lookup fails.

Iceberg views are also missed by the rest of the Iceberg view handling in this setup: DESCRIBE, SHOW CREATE TABLE, the schema binding mode, temp object checks, and others.

Fix

In SparkSessionCatalog.loadTable, the session catalog's result is not returned when it is a view that the Iceberg view catalog knows (viewExists). The original NoSuchTableException is rethrown instead, so ResolveViews resolves the view as an Iceberg view, the same as for the other catalogs. The extra viewExists call is only made when the session catalog returns a view. Hive views that aren't Iceberg views are still resolved by Spark, because HiveCatalog.viewExists is false for them.

Testing

  • New TestSessionCatalogHiveViews covers SparkSessionCatalog with type=hive, which TestViews doesn't cover. It has three tests:

    • column aliases over VALUES
    • the exact SQL from the issue, with WITH SCHEMA EVOLUTION
    • aliases with a column comment over a table

    All three fail without the fix with the error above.

  • Locally, I also ran TestViews with only the SparkSessionCatalog + Hive configuration as its parameter. That run isn't part of this PR.

    • Without the fix, 26 of its 86 tests fail.
    • With the fix, 25 of those pass and no new failures appear.
    • The remaining one, renameViewToDifferentTargetCatalog, fails with or without the fix. HiveCatalog.renameView rejects the three-part target identifier, which is a separate issue.
  • On Spark 4.2, TestSparkSessionCatalog, TestViews, TestViewCatalogCache, TestReferencedByViewChain, TestTempViewRefresh and TestSessionCatalogHiveViews pass.

Spark 3.5, 4.0 and 4.1 have the same code. I'll backport in a follow-up once this is reviewed.

…e as Iceberg views

The Hive catalog stores Iceberg views in the Hive Metastore as Hive views.
When SparkSessionCatalog fails to load such a view as an Iceberg table, it
falls back to the session catalog, which loads it as a Hive view. Spark then
resolves it as a v1 view before the ResolveViews rule of the Iceberg
extensions can, by name with the view's column names, which fails when the
view has column aliases:

  [INCOMPATIBLE_VIEW_SCHEMA_CHANGE] ... column c1 cannot be resolved.
  Expected 1 columns named c1 but got [].

Don't return a view of the session catalog that is an Iceberg view, so that
it is resolved as an Iceberg view, as it is with the other catalogs.

Closes apache#17288

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the spark label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spark: CREATE VIEW ... WITH SCHEMA EVOLUTION fails for VALUES queries during SELECT

2 participants