Skip to content

[Bug] Spark JOINs on masked partition keys fail with DPP #11013

Open

Description

@taoran92

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

2.1 & 2.2-SNAPSHOT

Compute Engine

Spark 3.5.8.

Minimal reproduce step

A query-authorized table has a masked partition key. Joining on that key fails when Spark applies dynamic partition pruning (DPP).

Use the existing SparkPartitionMetadataAuthTest.PolicySparkCatalog fixture from the paimon-spark-ut test classpath, with a fresh warehouse and these Spark configurations:

spark.sql.catalog.spark_test=org.apache.paimon.spark.SparkPartitionMetadataAuthTest$PolicySparkCatalog
spark.sql.catalog.spark_test.warehouse=file:///tmp/paimon-dpp-issue-warehouse
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions

The test.* table options below are specific to this fixture, not production configuration. They allow id and dt, apply the row filter id < 3, and mask dt as '***'.

CREATE TABLE spark_test.default.T (id INT, dt STRING)
USING paimon PARTITIONED BY (dt)
TBLPROPERTIES ('primary-key'='dt,id', 'bucket-key'='id', 'bucket'='1');

INSERT INTO spark_test.default.T VALUES
  (1, '20250810'), (2, '20250811'), (3, '20250810');

CREATE TABLE spark_test.default.D (dt STRING, k INT)
USING paimon TBLPROPERTIES ('bucket'='-1');

INSERT INTO spark_test.default.D VALUES ('***', 1), ('other', 0);

ALTER TABLE spark_test.default.T SET TBLPROPERTIES (
  'query-auth.enabled'='true',
  'test.auth-mode'='constant',
  'test.allowed'='id,dt'
);

SET spark.sql.optimizer.dynamicPartitionPruning.enabled=true;

SELECT l.id
FROM spark_test.default.T l
JOIN spark_test.default.D r ON l.dt = r.dt
WHERE r.k = 1
ORDER BY l.id;

Leave AQE and other DPP settings at their defaults.

What doesn't meet your expectations?

Expected: return IDs 1 and 2, since both authorized rows have the visible partition key '***'.

Actual: the query fails with:

java.lang.UnsupportedOperationException: A partition filter cannot be enforced on masked partition key 'dt': engines push partition predicates past the reader, so it would be matched against the raw partition value.

The same query returns IDs 1 and 2 with spark.sql.optimizer.dynamicPartitionPruning.enabled=false. With DPP enabled, the query fails whether AQE is enabled or disabled.

Anything else?

Root cause: Spark currently advertises masked partition keys as runtime-filterable. DPP derives predicates from visible join values, such as dt IN ('***'), but Paimon applies partition pruning to raw values. The core authorization check rejects this unsafe pushdown to prevent matching rows from being incorrectly pruned.

Related: #10837 (resolved by #10838) added authorization-aware partition metadata and corrected JOIN-inferred IS NOT NULL checks on masked keys.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions