Repository navigation
Description
Search before asking
- I searched in the issues and found nothing similar.
Paimon version
2.1 & 2.2-SNAPSHOT
Compute Engine
Spark 3.5.8.
Minimal reproduce step
A query-authorized table has a masked partition key. Joining on that key fails when Spark applies dynamic partition pruning (DPP).
Use the existing SparkPartitionMetadataAuthTest.PolicySparkCatalog fixture from the paimon-spark-ut test classpath, with a fresh warehouse and these Spark configurations:
spark.sql.catalog.spark_test=org.apache.paimon.spark.SparkPartitionMetadataAuthTest$PolicySparkCatalog
spark.sql.catalog.spark_test.warehouse=file:///tmp/paimon-dpp-issue-warehouse
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensionsThe test.* table options below are specific to this fixture, not production configuration. They allow id and dt, apply the row filter id < 3, and mask dt as '***'.
CREATE TABLE spark_test.default.T (id INT, dt STRING)
USING paimon PARTITIONED BY (dt)
TBLPROPERTIES ('primary-key'='dt,id', 'bucket-key'='id', 'bucket'='1');
INSERT INTO spark_test.default.T VALUES
(1, '20250810'), (2, '20250811'), (3, '20250810');
CREATE TABLE spark_test.default.D (dt STRING, k INT)
USING paimon TBLPROPERTIES ('bucket'='-1');
INSERT INTO spark_test.default.D VALUES ('***', 1), ('other', 0);
ALTER TABLE spark_test.default.T SET TBLPROPERTIES (
'query-auth.enabled'='true',
'test.auth-mode'='constant',
'test.allowed'='id,dt'
);
SET spark.sql.optimizer.dynamicPartitionPruning.enabled=true;
SELECT l.id
FROM spark_test.default.T l
JOIN spark_test.default.D r ON l.dt = r.dt
WHERE r.k = 1
ORDER BY l.id;Leave AQE and other DPP settings at their defaults.
What doesn't meet your expectations?
Expected: return IDs 1 and 2, since both authorized rows have the visible partition key '***'.
Actual: the query fails with:
java.lang.UnsupportedOperationException: A partition filter cannot be enforced on masked partition key 'dt': engines push partition predicates past the reader, so it would be matched against the raw partition value.
The same query returns IDs 1 and 2 with spark.sql.optimizer.dynamicPartitionPruning.enabled=false. With DPP enabled, the query fails whether AQE is enabled or disabled.
Anything else?
Root cause: Spark currently advertises masked partition keys as runtime-filterable. DPP derives predicates from visible join values, such as dt IN ('***'), but Paimon applies partition pruning to raw values. The core authorization check rejects this unsafe pushdown to prevent matching rows from being incorrectly pruned.
Related: #10837 (resolved by #10838) added authorization-aware partition metadata and corrected JOIN-inferred IS NOT NULL checks on masked keys.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity