Skip to content

[fix](catalog) Mask connector catalog secrets and the ALTER CATALOG audit text - #70338

Merged
morningman merged 4 commits into
apache:masterfrom
morningman:b2m-t38c-connector-sensitive-keys
Nov 17, 2026
Merged

morningman merged 4 commits into
apache:masterfrom
morningman:b2m-t38c-connector-sensitive-keys

Conversation

@morningman

@morningman morningman commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Background

SHOW CREATE CATALOG, SHOW CATALOG, the catalogs() table function and the audit text of CREATE CATALOG replace a property value with *XXX when DatasourcePrintableMap.isSensitiveKey matches its key. Until now the keys came from a list in fe-core and from the keys filesystem plugins report at startup. A connector plugin marks its secret properties sensitive = true, but that mark only masks the connector's own log lines. A connector tells the engine about its security-relevant properties through ConnectorProvider#securitySurface().

What problem does this PR solve?

Issue Number: None
Related PR: #69588 (added the connector security surface this extends), #45164 (introduced the unmasked ALTER CATALOG audit text), #27115 (masked it for the statement #45164 replaced), #59502 (added iceberg.jdbc.password), #54523 (added the Paimon REST DLF keys), #65730 (branch-4.1's Lance catalog, whose REST secrets the Lance port will declare on this surface), #70366 (fixed the shared Unicode SQL rewrite used by this PR's ALTER masking), #70327 (added MongoDB credentials to the engine masking floor)
PR Type: Bugfix

Problem Summary:

Catalog secrets that only a connector knows are on no masking list, so SHOW CREATE CATALOG and the other renderings print them in clear: iceberg.jdbc.password, paimon.jdbc.password, the Paimon REST DLF key pair, mc.access_key, and credentials passed through under trino. or adbc.. Separately, ALTER CATALOG ... SET PROPERTIES is written to the audit log unmasked, whatever the key. Each connector now declares its secrets on its security surface, taken from its sensitive marks, and the surface can also declare a namespace of secret keys for options passed through unread; Fluss's lake options were hard-coded in fe-core for that reason. The engine masks the union, with fe-core's own list kept as the floor. ALTER CATALOG is masked like CREATE CATALOG.

Details

  • Change map: 45 files, +1205 −81 over four commits: production code +384 −51, tests +807 −26, the connector README +13 −3, the plugin API version in fe/fe-connector/pom.xml (31.0 to 32.0). The first commit adds secret keys to the surface (reviewed alone before). The second does the rest, the third keeps a surface that fails in the connector's own ServiceLoader from stopping FE, and the fourth aligns the newly added MongoDB provider and tests ALTER masking with supplementary characters:
    1. Key namespaces on the SPI: withSensitiveKeyPrefix and SensitiveKeyPrefix.covers, which holds the matching rule; a collection form of withSensitiveKeys (ConnectorSecuritySurface).
    2. Engine: isSensitiveKey matches registered namespaces, the Fluss rule becomes one of them; ConnectorPluginManager registers them and reads every surface before it publishes a provider, skipping a provider whose surface throws.
    3. Connectors: every provider declares its secrets; MetaStoreProviders.sensitivePropertyKeys() gives the union of a plugin's metastore backends; IcebergJdbcMetaStoreProvider reports its keys like its siblings; the keytab paths lose their sensitive mark. MongoDB also declares its two annotated secrets, preserving its SSRF endpoint and the engine fallback.
    4. ALTER CATALOG audit masking (+16).
  • Frequency: Always. Create or alter a catalog that sets one of the keys above, then run SHOW CREATE CATALOG, SHOW CATALOG <name> or select * from catalogs(), or read the audit log after an ALTER CATALOG ... SET PROPERTIES.
  • Root cause: isSensitiveKey consulted only fe-core's list and the filesystem providers' keys. A connector's sensitive marks fed only toMaskedString; MetaStoreProvider.sensitivePropertyKeys() existed for this, but nothing called it, and the Iceberg JDBC backend did not implement it. For ALTER CATALOG, AlterCatalogPropertiesCommand does not implement NeedAuditEncryption and LogicalPlanBuilderForEncryption has no visitAlterCatalogProperties, so AuditLogHelper records the original text; the statement it replaced in [Enhancement] (nereids)implement alterCatalogPropertiesCommand in ner… #45164 was masked by [enhancement](audit) hide password and other sensitive information in log and audit log #27115.
  • Why existing tests missed it: Unit tests: DatasourcePrintableMapTest checks fe-core's own keys, the provider surface tests checked only the SSRF and driver_url keys, and EncryptSQLTest and RepositoryAuditEncryptionTest cover CREATE CATALOG, repositories and resources but not ALTER CATALOG. Regression: audit/test_external_catalog_credential_masking creates only Iceberg REST catalogs, whose keys are on fe-core's list, and never alters one.
  • Fix: The engine must not name a connector's properties, so the connectors declare their secrets, from the same sensitive marks that mask their logs; jdbc and es declare both spellings their legacy prefix allows, and trino and adbc declare the namespace they pass through, a key there being a secret when its rest names key, secret, password, token or credential (also authorization for ADBC). fe-core keeps the connector keys it lists, so a catalog whose plugin is not loaded masks what it masked before. A provider whose securitySurface() throws at startup is skipped, as a filesystem provider is, instead of stopping FE; that includes the ServiceConfigurationError a ServiceLoader throws when a descriptor names a backend it cannot load, which is how the Hive, Iceberg and Paimon surfaces fail, since they read their metastore backends' keys through one.
  • Other paths and branches: Fixed here: every rendering above and FE's statement logging, which share isSensitiveKey and NeedAuditEncryption. Not covered, since a key name cannot express them safely: raw Iceberg SDK keys copied through (token, credential), Paimon options re-keyed from paimon.*, Hadoop fs.* keys, and secrets inside otherwise unmarked URLs. The ALTER CATALOG gap is on branch-4.0, 4.1 and 4.2 (from [Enhancement] (nereids)implement alterCatalogPropertiesCommand in ner… #45164); branch-3.1 still uses the masked statement. branch-4.1 and 4.2 also print the connector keys above in clear; without the connector SPI their fix differs and is tracked separately.
  • Behavior change: The namespaces also mask non-secrets whose names carry those words, such as trino.hive.s3.aws-access-key or a keystore path. Keytab paths, which no release masks, are no longer masked in connector log lines: marked sensitive, hadoop.kerberos.keytab would be masked in every storage vault and resource.

Testing

The original local runs (macOS) used master f5c993f2c94 plus this change:

  • Regression audit/test_external_catalog_credential_masking, extended with an Iceberg JDBC catalog: SHOW CREATE CATALOG masks iceberg.jdbc.password, and the audit texts of its CREATE CATALOG and of an ALTER CATALOG that sets the password and s3.secret_key hold only *XXX. Against the local cluster's FE without this change it fails at the first new check, SHOW CREATE CATALOG printing the password; with a FE built from this change it passes.
  • fe-connector-spi: all 160 tests, with the refreshed surface baseline (+4 lines); the metastore modules: all tests; ten connector modules: 29 classes, 511 tests, including each connector's surface test, which checks real key names.
  • fe-core: 26 classes, 341 tests, including ConnectorPluginManagerTest, DatasourcePrintableMapTest, EncryptSQLTest and the audit and plugin gate tests; 14 of them again after the last change to registration, and 7 classes, 76 tests, after the third commit. Its two new ConnectorPluginManagerTest cases, a plugin directory whose lib/ holds a jar with a service descriptor naming a missing metastore-style backend (the plugin next to it must still load) and the same descriptor on the classpath, fail without the third commit with that ServiceConfigurationError escaping loadPlugins and loadBuiltins, and pass with it.
  • Twenty-two mutations, each caught by its test: for example, reverting either half of the ALTER CATALOG fix (EncryptSQLTest sees the statement in clear), matching a marker anywhere in the key, an empty metastore union, re-marking a keytab, or dropping ServiceConfigurationError from either startup catch (each fails only its own new case).

Rebased onto master 93c16ee2ca1 on 2026-11-16. Post-rebase local verification:

  • bash run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest', with CUSTOM_MVN appending -Dcheckstyle.skip=false: BUILD SUCCESS across all 90 reactor modules; all 141 selected tests passed across 22 classes, including the inherited supplementary-character audit cases. Checkstyle reported no violations.
  • git diff --check upstream-apache/master...HEAD: passed. git range-diff confirms the replayed patches are unchanged apart from adapting the connector API version and its test to 32.0.
  • Regression tests and a complete BE/FE build were not rerun locally after this rebase; run buildall is requested in CI.

Follow-up local verification on the rebased branch plus the MongoDB declaration and audit tests (2026-11-16):

  • run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,org.apache.doris.connector.mongodb.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest', with Checkstyle enabled: all 176 selected tests in 28 classes passed (SPI 93, MongoDB 35, fe-core 48), with no skipped tests; the 90-module reactor completed successfully.
  • The strengthened MongoProviderTest fails without the provider declaration (expected the two MongoDB secret keys, received an empty set), and passes with it. A separate targeted revert confirms the assertion catches the missing declaration.
  • The new ALTER CATALOG assertions cover supplementary characters within a secret, in a preceding SQL comment, and in a retained non-secret value, with two masked properties. EncryptSQLTest#testEncryption passes on the candidate and fails at the new ALTER assertion when only BaseViewInfo.rewriteSql is restored to its pre-[fix](nereids) Fix SQL masking and view rewrites after supplementary characters #70366 implementation: the audit text has an extra trailing fragment.
  • Full cluster regression and a standalone FE/BE build were not rerun for this follow-up.

Notes for Reviewers

  • SensitiveKeyPrefix.covers lives on the SPI so that a connector's test can check its declaration against real key names and the engine applies the same rule; the Fluss floor in fe-core uses it too.
  • The entries both fe-core and a connector declare are deliberate: the floor is frozen, as the comment in DatasourcePrintableMap says. MongoDB was already masked by [feature](mongodb) Add native catalog, reader and BSON SerDe #70327; its added declaration makes the provider follow the same contract as the other connectors.
  • The connector plugin API advances from 31.0 to 32.0 for the added security-surface methods. Third-party connector plugins built against 31.x must be rebuilt; the in-tree plugins are built with FE. Hudi declares nothing: it owns no catalog properties.
  • The base includes [fix](nereids) Fix SQL masking and view rewrites after supplementary characters #70366, so ALTER CATALOG masking uses the shared rewrite that converts parser code-point spans to Java string offsets, including supplementary characters.

Release note

SHOW CREATE CATALOG, SHOW CATALOG, the catalogs() table function and the audit log now mask iceberg.jdbc.password, paimon.jdbc.password, paimon.rest.dlf.access-key-id, paimon.rest.dlf.access-key-secret, mc.access_key, and the trino.* and adbc.* properties whose names carry key, secret, password, token or credential (authorization too for adbc.*). ALTER CATALOG ... SET PROPERTIES statements are masked in the audit log like CREATE CATALOG. Third-party connector plugins built against API 31.x must be rebuilt for 32.0.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Ready

AI Disclosure

  • Tool: Claude Code (Claude Opus 5.5); OpenAI Codex
  • Generated: Claude Code wrote the original code, tests, and description. Codex rebased the commits, adapted the connector API version and its test, added the MongoDB declaration and ALTER CATALOG supplementary-character tests, and updated this description.
  • Human review: not stated
  • Prompts: The author requested a generic connector surface, prefix rules, declarations for every connector, and masking of their union with the engine floor. They removed sensitive marks from keytab paths, added ALTER CATALOG masking, and fixed startup failure from a connector's ServiceLoader. After review, they chose a separate shared Unicode fix, then requested MongoDB alignment and supplementary-character audit coverage here.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review recovery stopped: shared time budget or helper timeout exhausted
Workflow run: https://github.com/apache/doris/actions/runs/42183157017

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28644 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44ed6b1ec5387a23cee6b554c88a1bf137f2d137, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17594	5134	5076	5076
q2	2143	306	254	254
q3	10247	1426	844	844
q4	4676	467	346	346
q5	7433	817	543	543
q6	193	191	155	155
q7	750	794	607	607
q8	9267	1353	1083	1083
q9	5555	4231	4222	4222
q10	6827	1292	1049	1049
q11	427	250	228	228
q12	651	401	280	280
q13	18115	3606	3084	3084
q14	287	274	243	243
q15	q16	723	708	671	671
q17	1220	762	581	581
q18	7112	6217	6131	6131
q19	1115	959	594	594
q20	383	347	229	229
q21	5397	2169	2289	2169
q22	359	296	255	255
Total cold run time: 100474 ms
Total hot run time: 28644 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5639	5627	5601	5601
q2	695	572	538	538
q3	5183	5404	4991	4991
q4	2024	2176	1546	1546
q5	5301	5199	5901	5199
q6	293	198	146	146
q7	2391	2123	1844	1844
q8	3135	2913	2792	2792
q9	8153	8062	8091	8062
q10	3704	3670	3309	3309
q11	611	436	416	416
q12	699	682	556	556
q13	3656	4009	3264	3264
q14	309	303	283	283
q15	q16	716	747	685	685
q17	8436	7621	7106	7106
q18	12959	12203	12969	12203
q19	913	801	791	791
q20	2151	2143	1923	1923
q21	5710	4765	4667	4667
q22	517	482	408	408
Total cold run time: 73195 ms
Total hot run time: 66330 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 154111 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44ed6b1ec5387a23cee6b554c88a1bf137f2d137, data reload: false

query5	4314	528	407	407
query6	425	205	186	186
query7	4809	425	233	233
query8	323	168	154	154
query9	8800	3190	3200	3190
query10	501	315	260	260
query11	6047	3877	3588	3588
query12	149	91	88	88
query13	1246	434	343	343
query14	6119	3929	3654	3654
query14_1	3430	3422	3395	3395
query15	216	207	183	183
query16	949	417	398	398
query17	874	665	530	530
query18	2427	434	328	328
query19	203	169	163	163
query20	82	81	82	81
query21	206	124	109	109
query22	12998	12964	12869	12869
query23	13371	12795	12484	12484
query23_1	12559	12522	12489	12489
query24	7366	821	469	469
query24_1	481	479	472	472
query25	522	400	342	342
query26	1265	263	136	136
query27	2759	440	275	275
query28	4611	2044	2043	2043
query29	1569	587	444	444
query30	300	215	180	180
query31	1005	923	812	812
query32	149	89	88	88
query33	522	294	234	234
query34	907	860	508	508
query35	724	749	642	642
query36	804	826	753	753
query37	128	93	83	83
query38	1801	1731	1631	1631
query39	800	798	752	752
query39_1	746	734	740	734
query40	214	121	97	97
query41	65	62	60	60
query42	93	82	85	82
query43	345	361	309	309
query44	1301	713	700	700
query45	187	177	160	160
query46	845	964	560	560
query47	2856	2873	2778	2778
query48	294	312	220	220
query49	581	421	301	301
query50	700	280	242	242
query51	10723	10453	10394	10394
query52	79	79	71	71
query53	186	204	155	155
query54	253	191	185	185
query55	72	65	62	62
query56	221	228	219	219
query57	1704	1645	1539	1539
query58	290	270	257	257
query59	2234	2270	1963	1963
query60	282	243	233	233
query61	151	151	146	146
query62	380	328	280	280
query63	188	158	161	158
query64	2741	941	777	777
query65	3329	3262	3261	3261
query66	1755	436	302	302
query67	22701	22696	22574	22574
query68	2960	1030	632	632
query69	388	288	245	245
query70	856	769	767	767
query71	303	237	218	218
query72	2838	2539	2514	2514
query73	529	528	296	296
query74	4610	4537	4368	4368
query75	2355	2286	1917	1917
query76	2343	1008	573	573
query77	337	367	277	277
query78	9818	9836	9236	9236
query79	899	842	532	532
query80	538	420	328	328
query81	519	323	268	268
query82	205	137	110	110
query83	195	200	183	183
query84	285	116	105	105
query85	756	421	374	374
query86	292	240	232	232
query87	1984	2004	1796	1796
query88	3717	2768	2755	2755
query89	319	276	246	246
query90	1914	173	167	167
query91	163	159	133	133
query92	104	91	88	88
query93	1007	995	581	581
query94	562	312	285	285
query95	571	332	290	290
query96	683	497	241	241
query97	2438	2438	2343	2343
query98	161	151	149	149
query99	759	775	669	669
Total cold run time: 236310 ms
Total hot run time: 154111 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44ed6b1ec5387a23cee6b554c88a1bf137f2d137, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.11	0.09
query5	0.24	0.25	0.25
query6	1.18	0.71	0.69
query7	0.04	0.01	0.01
query8	0.05	0.03	0.04
query9	0.39	0.35	0.36
query10	0.58	0.57	0.60
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.50	0.50	0.48
query14	1.02	1.00	1.02
query15	0.63	0.61	0.60
query16	0.36	0.34	0.36
query17	1.16	1.11	1.11
query18	0.21	0.19	0.20
query19	2.15	1.97	2.00
query20	0.02	0.02	0.01
query21	15.46	0.19	0.14
query22	4.80	0.06	0.05
query23	16.29	0.33	0.12
query24	1.10	0.21	0.17
query25	0.11	0.05	0.05
query26	0.74	0.22	0.16
query27	0.04	0.03	0.04
query28	3.64	0.86	0.37
query29	12.50	4.15	3.26
query30	0.28	0.13	0.10
query31	2.77	0.56	0.34
query32	3.22	0.61	0.50
query33	3.22	3.17	3.19
query34	15.55	4.02	3.43
query35	3.37	3.35	3.34
query36	0.56	0.43	0.42
query37	0.10	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.04	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 95.07 s
Total hot run time: 24.13 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

@morningman morningman changed the title [improvement](catalog) Let connector plugins declare secret catalog properties to mask [fix](catalog) Mask connector catalog secrets and the ALTER CATALOG audit text Nov 6, 2026
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 42 changed paths at b0bfd3c. One new P2 finding: eager security-surface discovery does not isolate nested metastore ServiceLoader failures, so one broken connector can prevent FE startup. No P0/P1 findings or existing blocking comments were confirmed.

Critical checkpoint conclusions:

Checkpoint Conclusion
Goal and proof Connector declarations reach SHOW CREATE CATALOG, SHOW CATALOG, catalogs(), and CREATE/ALTER audit/log masking. Unit and audit regression coverage was inspected; execution was outside this static review.
Scope and reuse Focused rendering change using existing annotations, the security surface, and NeedAuditEncryption; no connector property parsing added to generic engine paths.
Concurrency and locks Normal registration runs before catalog replay/request access. New prefix reads use copy-on-write snapshots and immutable rules; no new threads, blocking IO under metadata locks, or lock-order changes.
Lifecycle and static initialization Provider-owned loaders and normal package dependencies are consistent. The inline P2 covers uncaught ServiceConfigurationError during newly eager metastore initialization.
Configuration No new configuration or dynamic-update requirement.
Compatibility Connector API major 31.0 matches the resource, manifests and test; plugin surface baseline includes additions. Storage and FE/BE protocols are unchanged.
Thrift and protobuf No changes.
Parallel paths Reviewed exact aliases, metastore flavors, pass-through prefixes, all three catalog renderings, CREATE/ALTER audit, statement logging, forwarding and multi-statement dispatch. Legacy masking floor is preserved.
Conditional logic Required ALTER property spans follow from grammar. Case-insensitive prefix/suffix rules match the declared behavior; keytab properties contain paths.
Test coverage Positive/negative aliases, namespace matching, immutability, registration/isolation and persisted audit checks are present. The new exception fixture misses ServiceConfigurationError, as noted inline.
Changed expected results No result files changed; added audit assertions use known placeholders and check masking plus retained nonsecret values. No generated test result is claimed.
Observability Skip logs identify providers/plugins; the reported exception currently bypasses them. No additional metrics are needed for this startup/rendering change.
Transactions and persistence Catalog journaling, replay, failover and persisted credential values are unchanged; startup registration also applies masking to historical catalogs.
Data writes and atomicity Table writes, committed-data visibility, delete bitmaps and crash recovery are unaffected.
FE/BE variables None added.
Performance Exact-key lookup and a small startup-defined prefix scan remain outside row-processing hot paths; no material regression established.
Other concerns No further substantiated issue after the final recheck.

No additional user focus was supplied. Existing inline threads were empty and the finding was deduplicated across reviewers. The PR's disclosed limits for arbitrary SDK/re-keyed/fs.* options and secrets embedded in URLs remain outside its declared masking coverage. Review coverage is complete with no unresolved candidates. Review converged after two rounds.

Validation was static source, test, dependency and control-flow inspection only. No builds, tests, benchmarks, runtime plugin loading or source edits were performed.

Review converged after 2 round(s).

ConnectorSecuritySurface securitySurface;
try {
securitySurface = provider.securitySurface();
} catch (RuntimeException | LinkageError e) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Catch ServiceConfigurationError during eager surface discovery

An otherwise valid connector directory with a stale metastore service descriptor, or a backend whose constructor fails, now prevents FE startup. Iceberg/Paimon/Hive securitySurface() calls MetaStoreProviders.sensitivePropertyKeys(), whose static initializer runs ServiceLoader; those lookup/instantiation failures throw ServiceConfigurationError, an Error outside both RuntimeException and LinkageError. It escapes this catch through Env.initConnectorPluginManager() before image loading, defeating the documented per-plugin skip behavior. Previously these nested backends were discovered during catalog validation/access rather than directory registration. Catch ServiceConfigurationError here and at the new builtin surface boundary, and cover a failing nested loader; the added IllegalStateException fixture does not exercise this case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ae2cebe. Confirmed against the code: the Hive, Iceberg and Paimon surfaces call MetaStoreProviders.sensitivePropertyKeys(), whose static initializer iterates a ServiceLoader over the plugin's jars. A descriptor naming a backend that cannot be loaded throws ServiceConfigurationError, an Error a static initializer rethrows unwrapped, so it escaped this RuntimeException | LinkageError catch and the builtin path's RuntimeException catch, and Env.initConnectorPluginManager() catches nothing: a stale jar in a connector's lib/ stopped FE at startup, where before this PR only that connector's catalogs failed. Both catches now include ServiceConfigurationError (the builtin one matches the construction catch above it), so the provider is skipped and logged like any other surface failure.

ConnectorPluginManagerTest#pluginWhoseSecuritySurfaceFailsInItsOwnServiceLoaderIsSkippedNotFatal (a plugin directory whose lib/ jar names a missing backend; the plugin next to it still loads) and #classpathProviderWhoseSecuritySurfaceFailsInItsOwnServiceLoaderIsSkippedNotFatal fail on b0bfd3c with that error and pass now; dropping it from either catch fails only that path's case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The mechanism is right, but it exists only on a broken test classpath, so I'm keeping the builtin catch as it is.

In production Hive, Iceberg and Paimon are never builtins: fe-core depends on them in test scope only, and FE loads them from plugins/connector/, one loader per plugin. MetaStoreProviders is not on FE's classpath (fe-core does not depend on fe-connector-metastore-spi), so each plugin defines its own copy, a failed static initializer cannot reach another provider, and the directory path's catch already includes LinkageError.

The two catches follow FileSystemPluginManager: its loadBuiltins catches RuntimeException around a provider's sensitivePropertyKeys(), and its directory path adds LinkageError. On the classpath, a descriptor naming a missing backend is a build defect, and a LinkageError there already stops startup elsewhere, for example from ServiceLoader's hasNext() when a provider class does not link. Here the first provider is skipped and logged, and the second stops startup with a NoClassDefFoundError naming the class whose initializer failed, which points at the defect.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28677 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0bfd3c1b10d83e6a4598af164544d695d8dc3ce, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17572	5202	5057	5057
q2	2120	304	245	245
q3	10267	1408	813	813
q4	4678	457	338	338
q5	7438	803	517	517
q6	187	189	159	159
q7	755	795	592	592
q8	9258	1367	1102	1102
q9	5542	4178	4165	4165
q10	6805	1298	1017	1017
q11	434	244	227	227
q12	630	398	273	273
q13	18128	3657	3069	3069
q14	275	276	241	241
q15	q16	727	718	668	668
q17	1204	759	586	586
q18	6974	6176	6100	6100
q19	1110	975	614	614
q20	402	340	221	221
q21	5473	2482	2399	2399
q22	382	320	274	274
Total cold run time: 100361 ms
Total hot run time: 28677 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5958	5920	6081	5920
q2	744	606	602	602
q3	5584	5830	5336	5336
q4	1999	2125	1541	1541
q5	5757	5793	5666	5666
q6	238	211	146	146
q7	2250	1977	1756	1756
q8	2956	2651	2721	2651
q9	7949	7488	7437	7437
q10	3611	3651	3116	3116
q11	570	404	384	384
q12	670	673	511	511
q13	3328	3569	3012	3012
q14	283	292	260	260
q15	q16	655	682	621	621
q17	7659	6993	6869	6869
q18	12983	12193	12937	12193
q19	958	838	821	821
q20	2127	2142	1906	1906
q21	5651	4576	4629	4576
q22	484	458	404	404
Total cold run time: 72414 ms
Total hot run time: 65728 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 153854 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0bfd3c1b10d83e6a4598af164544d695d8dc3ce, data reload: false

query5	4320	519	387	387
query6	430	223	197	197
query7	4811	427	232	232
query8	315	160	148	148
query9	8790	3187	3183	3183
query10	465	312	260	260
query11	6056	3883	3568	3568
query12	152	94	89	89
query13	1262	436	326	326
query14	6139	3918	3756	3756
query14_1	3412	3425	3425	3425
query15	206	199	181	181
query16	916	432	391	391
query17	851	652	527	527
query18	2414	414	304	304
query19	191	168	137	137
query20	84	83	82	82
query21	210	129	105	105
query22	12960	12927	12820	12820
query23	13540	13010	12467	12467
query23_1	12601	12711	12523	12523
query24	7351	831	479	479
query24_1	496	491	480	480
query25	528	426	362	362
query26	1258	261	140	140
query27	2762	437	278	278
query28	4624	2044	2047	2044
query29	1586	597	475	475
query30	294	220	181	181
query31	1023	925	819	819
query32	143	94	90	90
query33	529	304	234	234
query34	928	855	506	506
query35	729	754	654	654
query36	820	800	750	750
query37	129	95	88	88
query38	1763	1739	1639	1639
query39	801	793	743	743
query39_1	729	722	743	722
query40	228	119	98	98
query41	71	68	68	68
query42	89	91	86	86
query43	341	359	315	315
query44	1321	708	713	708
query45	183	184	163	163
query46	845	964	566	566
query47	2638	2933	2840	2840
query48	338	313	225	225
query49	575	400	296	296
query50	681	274	211	211
query51	10791	10405	10645	10405
query52	83	80	69	69
query53	186	212	156	156
query54	224	215	189	189
query55	77	64	62	62
query56	236	203	227	203
query57	1656	1626	1587	1587
query58	289	268	262	262
query59	2154	2219	2033	2033
query60	276	250	217	217
query61	150	146	147	146
query62	390	340	282	282
query63	184	162	166	162
query64	2738	962	761	761
query65	3326	3250	3279	3250
query66	1766	420	291	291
query67	22730	22538	22446	22446
query68	3092	1002	616	616
query69	417	292	261	261
query70	836	770	774	770
query71	308	243	212	212
query72	2800	2584	2274	2274
query73	520	525	295	295
query74	4617	4500	4317	4317
query75	2338	2293	1942	1942
query76	2340	1006	606	606
query77	355	374	270	270
query78	10020	9629	9132	9132
query79	974	843	529	529
query80	1107	421	332	332
query81	582	319	263	263
query82	588	138	113	113
query83	313	199	193	193
query84	304	123	101	101
query85	854	416	377	377
query86	379	240	227	227
query87	2018	1969	1873	1873
query88	3788	2783	2776	2776
query89	323	285	242	242
query90	1930	171	167	167
query91	166	150	124	124
query92	88	92	88	88
query93	1005	991	577	577
query94	564	284	267	267
query95	566	319	301	301
query96	690	490	247	247
query97	2497	2429	2333	2333
query98	160	147	153	147
query99	767	764	673	673
Total cold run time: 238098 ms
Total hot run time: 153854 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.8 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0bfd3c1b10d83e6a4598af164544d695d8dc3ce, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.08	0.08
query4	1.62	0.09	0.10
query5	0.23	0.24	0.24
query6	1.17	0.69	0.67
query7	0.04	0.01	0.00
query8	0.05	0.03	0.04
query9	0.39	0.34	0.31
query10	0.61	0.57	0.57
query11	0.20	0.14	0.14
query12	0.17	0.15	0.15
query13	0.49	0.48	0.48
query14	1.03	1.01	1.00
query15	0.62	0.62	0.61
query16	0.34	0.33	0.34
query17	1.14	1.10	1.12
query18	0.22	0.17	0.17
query19	2.08	2.00	1.93
query20	0.02	0.02	0.01
query21	15.42	0.18	0.13
query22	4.89	0.06	0.05
query23	16.29	0.32	0.13
query24	1.12	0.25	0.16
query25	0.10	0.05	0.03
query26	0.73	0.20	0.15
query27	0.04	0.03	0.03
query28	3.75	0.81	0.35
query29	12.50	4.19	3.24
query30	0.28	0.13	0.10
query31	2.77	0.54	0.32
query32	3.22	0.59	0.50
query33	3.18	3.20	3.18
query34	15.43	4.00	3.37
query35	3.34	3.32	3.33
query36	0.57	0.43	0.42
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 94.93 s
Total hot run time: 23.8 s

@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 43 changed paths at ae2cebe. One new P1 finding: the newly enabled ALTER CATALOG masking uses code-point token positions as UTF-16 substring offsets and leaves plaintext credential suffixes in audit and FE statement logs. This prevents the intended complete masking fix for accepted supplementary-Unicode SQL.

The inspected flow is:

provider -> securitySurface -> startup masking union -> SHOW CREATE / SHOW CATALOG / catalogs()
ALTER CATALOG -> NeedAuditEncryption -> property spans -> rewriteSql -> audit / FE logs

Critical checkpoint conclusions:

Checkpoint Conclusion
Goal and proof Connector-declared secrets reach the three catalog renderings and CREATE/ALTER logging. The inline P1 identifies the Unicode gap in the new ALTER path. Added unit and audit regression cases cover ASCII behavior; no execution is claimed.
Scope, clarity and reuse Focused masking change reusing property annotations, the existing security surface and NeedAuditEncryption. Generic engine paths gain no connector-specific property parsing.
Concurrency and locks Production registration occurs before image/replay/catalog access. Prefix rules use copy-on-write snapshots and immutable descriptors. No new thread, metadata-lock IO, nested lock order or supported production read/write race was identified.
Lifecycle and static initialization Metastore discovery explicitly uses the defining plugin loader. Directory failures discard/close handles before routing publication; providers and masking descriptors otherwise live for FE lifetime. Both startup ServiceConfigurationError boundaries now catch the failure reported in the existing thread.
Configuration and dynamic updates No new configuration or dynamic-update requirement.
Compatibility and rolling upgrades SPI major and test pin both move from 30.0 to 31.0; the plugin API baseline records the additions. Legacy masking floor is retained. No durable format or FE/BE protocol changes.
Thrift and protobuf No changes.
Parallel paths Checked aliases, metastore flavors, pass-through prefixes, all three catalog renderings, CREATE/ALTER audit, FE statement logging, forwarding and multi-statement dispatch. Shared rewrite index units cause the inline finding.
Conditional logic ALTER property lists are mandatory in the grammar. Namespace matching is case-insensitive and checks only the suffix after the declared prefix. Keytab annotations removed here name paths; keytab-content masking remains.
Test coverage Inspected positive/negative aliases, prefix boundaries, immutability, admitted/refused providers, ADBC discovery isolation, actual nested ServiceLoader failure fixtures and persisted audit assertions. Supplementary Unicode values/comments are missing.
Changed expected results No golden result files changed. Added assertions match ASCII masking and retained nonsecret values; they do not establish Unicode correctness. Result generation and author-reported test runs were not verified by execution.
Observability Skip logs identify provider/plugin failures. The Unicode leak is a successful string rewrite, so existing exception fallback cannot detect it. No additional metric is needed for this startup/rendering change.
Transactions and persistence Catalog journaling/replay/failover and stored credentials are unchanged. Startup registration applies masks to historical catalogs.
Data writes and atomicity Table writes, committed-data visibility, version consistency, delete bitmaps and crash recovery are unaffected. No BE allocation/ownership change.
FE/BE variables None added.
Performance Small startup-defined prefix scans and exact-key lookups run in rendering paths; no new IO or row-processing hot-path work. No material performance defect substantiated.
Other concerns No further substantiated issue after full-scope and missed-case rechecks.

Existing thread 4492689479 is independently confirmed addressed at this head: both relevant catches include ServiceConfigurationError, and the new classpath/directory fixtures exercise a missing nested backend. It is not duplicated. No existing P0/P1 comments remain to carry forward. No additional user review focus was supplied.

The PR's disclosed limits for arbitrary raw/re-keyed SDK options, fs.* options and secrets embedded in URLs remain outside its declared masking coverage. Static review coverage is complete with no unresolved candidates or risk questions. Review converged after two rounds.

Validation was static source, test, dependency and control-flow inspection, with a standalone character-index calculation for the inline example. No Doris builds/tests, benchmarks, runtime plugin loading, submodule initialization or source edits were performed.

Review converged after 2 round(s).

public LogicalPlan visitAlterCatalogProperties(DorisParser.AlterCatalogPropertiesContext ctx) {
DorisParser.PropertyItemListContext properties = ctx.propertyItemList();
encryptProperty(visitPropertyItemList(properties), properties.start.getStartIndex(),
properties.stop.getStopIndex());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Convert code-point spans before slicing the original SQL

The new ALTER masking path passes ANTLR offsets directly to BaseViewInfo.rewriteSql, which slices a Java String using UTF-16 offsets. CaseInsensitiveStream.fromString uses a code-point stream whenever the SQL contains a supplementary character, so the replacement ends early and copies part of the original credential back into the result. For example, ALTER CATALOG ctl SET PROPERTIES("password"="😀😀😀😀😀😀😀secret") produces ALTER CATALOG ctl SET PROPERTIES("password" = "*XXX"secret"): the entire ASCII suffix remains in the audit and FE statement logs, without an exception to trigger their fallback. Supplementary characters in an earlier SQL comment can expose an otherwise ASCII password too. Convert both token boundaries to UTF-16 positions (including the exclusive end) before rewriting, and cover Unicode values/comments in the masking tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed. This is a defect the engine already had, not one this PR introduces: when a statement contains a supplementary character, the parser's token positions count code points, while BaseViewInfo.rewriteSql cuts the Java string by UTF-16 offsets. Every masking visitor in LogicalPlanBuilderForEncryption shares it (CREATE CATALOG, repositories, SET PASSWORD and the rest), and so does the rewriting of view and MTMV definitions; the ALTER CATALOG masking added here goes through the same rewriteSql.

It is fixed separately in #70366, which converts the positions at all five places that cut the original SQL and adds cases with emoji in values, in comments before the masked span, and across several replacements in one statement. Once that is merged, the ALTER CATALOG path in this PR gets the fix without a change here.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28298 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae2cebe750f695b3a64abf1893c65982a67cf29f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17586	5123	5310	5123
q2	2072	290	242	242
q3	10287	1365	814	814
q4	4678	465	336	336
q5	7438	785	519	519
q6	183	180	150	150
q7	728	778	594	594
q8	9241	1325	1063	1063
q9	5546	4163	4195	4163
q10	6814	1293	999	999
q11	418	242	219	219
q12	637	394	277	277
q13	18083	3552	2956	2956
q14	278	274	240	240
q15	q16	703	702	661	661
q17	1188	758	568	568
q18	6982	6139	6109	6109
q19	1105	950	587	587
q20	384	336	221	221
q21	5391	2207	2270	2207
q22	342	292	250	250
Total cold run time: 100084 ms
Total hot run time: 28298 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5593	5670	5608	5608
q2	703	565	532	532
q3	5218	5405	4955	4955
q4	1991	2146	1521	1521
q5	5255	5169	5568	5169
q6	251	202	155	155
q7	2281	2060	1906	1906
q8	3110	2741	2647	2647
q9	8049	8001	7951	7951
q10	3692	3894	3212	3212
q11	630	446	401	401
q12	716	693	536	536
q13	3498	3824	3334	3334
q14	319	311	295	295
q15	q16	706	702	646	646
q17	8333	7091	6853	6853
q18	12999	12149	12839	12149
q19	896	794	804	794
q20	2139	2122	1935	1935
q21	5621	4559	4666	4559
q22	506	434	405	405
Total cold run time: 72506 ms
Total hot run time: 65563 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 153663 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae2cebe750f695b3a64abf1893c65982a67cf29f, data reload: false

query5	4305	516	400	400
query6	422	203	189	189
query7	4803	430	238	238
query8	313	163	147	147
query9	8774	3212	3207	3207
query10	483	313	269	269
query11	6037	3853	3532	3532
query12	150	93	89	89
query13	1262	459	348	348
query14	6122	3950	3657	3657
query14_1	3410	3394	3395	3394
query15	204	202	186	186
query16	941	438	413	413
query17	864	657	570	570
query18	2423	436	293	293
query19	192	158	122	122
query20	83	83	78	78
query21	207	122	102	102
query22	12884	12884	12739	12739
query23	13412	12827	12467	12467
query23_1	12549	12430	12531	12430
query24	7362	804	474	474
query24_1	474	471	477	471
query25	508	387	333	333
query26	1260	250	133	133
query27	2770	437	268	268
query28	4645	2068	2051	2051
query29	1549	576	443	443
query30	289	217	179	179
query31	1016	924	802	802
query32	144	88	86	86
query33	508	296	220	220
query34	870	839	489	489
query35	724	748	636	636
query36	819	852	742	742
query37	120	91	99	91
query38	1766	1700	1662	1662
query39	781	778	727	727
query39_1	725	729	743	729
query40	214	112	94	94
query41	65	63	59	59
query42	87	82	80	80
query43	338	357	315	315
query44	1303	701	706	701
query45	178	177	162	162
query46	837	977	568	568
query47	3018	2926	2577	2577
query48	317	318	235	235
query49	582	407	306	306
query50	697	278	216	216
query51	10806	10732	10479	10479
query52	79	84	70	70
query53	195	210	153	153
query54	240	246	197	197
query55	69	71	68	68
query56	242	202	214	202
query57	1648	1647	1559	1559
query58	281	277	257	257
query59	2098	2143	2008	2008
query60	287	240	204	204
query61	149	144	144	144
query62	382	329	279	279
query63	187	156	159	156
query64	2722	902	764	764
query65	3310	3271	3229	3229
query66	1765	428	290	290
query67	22618	22841	22700	22700
query68	2945	1017	628	628
query69	383	289	250	250
query70	863	759	779	759
query71	304	233	215	215
query72	2657	2686	2438	2438
query73	524	536	291	291
query74	4597	4528	4345	4345
query75	2349	2265	1901	1901
query76	2327	1012	600	600
query77	353	381	282	282
query78	9756	9650	9205	9205
query79	1011	864	541	541
query80	754	448	365	365
query81	554	307	264	264
query82	585	144	120	120
query83	301	215	189	189
query84	321	127	105	105
query85	939	440	366	366
query86	388	241	226	226
query87	1988	1943	1818	1818
query88	3727	2776	2790	2776
query89	333	284	243	243
query90	1897	167	164	164
query91	160	149	124	124
query92	99	92	87	87
query93	1006	1007	580	580
query94	543	303	291	291
query95	559	386	305	305
query96	703	490	237	237
query97	2485	2445	2342	2342
query98	165	154	147	147
query99	768	767	653	653
Total cold run time: 236925 ms
Total hot run time: 153663 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.08 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae2cebe750f695b3a64abf1893c65982a67cf29f, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.04	0.05
query3	0.26	0.09	0.08
query4	1.60	0.10	0.10
query5	0.25	0.23	0.24
query6	1.16	0.68	0.68
query7	0.04	0.01	0.00
query8	0.06	0.03	0.03
query9	0.40	0.35	0.35
query10	0.56	0.58	0.63
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.50	0.49	0.48
query14	1.02	1.00	1.01
query15	0.61	0.59	0.60
query16	0.34	0.34	0.34
query17	1.16	1.16	1.16
query18	0.21	0.17	0.18
query19	2.13	2.04	2.02
query20	0.02	0.01	0.01
query21	15.41	0.20	0.13
query22	4.83	0.05	0.05
query23	16.30	0.30	0.12
query24	1.07	0.23	0.19
query25	0.10	0.04	0.04
query26	0.84	0.21	0.16
query27	0.04	0.03	0.03
query28	3.73	0.78	0.35
query29	12.53	4.16	3.23
query30	0.28	0.12	0.10
query31	2.77	0.55	0.32
query32	3.23	0.59	0.51
query33	3.28	3.24	3.26
query34	15.32	3.99	3.40
query35	3.34	3.30	3.31
query36	0.54	0.43	0.42
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 94.92 s
Total hot run time: 24.08 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 94.44% (34/36) 🎉
Increment coverage report
Complete coverage report

morningman added a commit to morningman/doris that referenced this pull request Nov 13, 2026
…characters

### Background

Several FE features rewrite the text of a statement by cutting the original SQL string at token positions from the parser. Audit masking replaces credential properties and passwords with `*XXX` before the statement reaches `fe.audit.log`, `__internal_schema.audit_log` and the FE log; CREATE / ALTER VIEW and CREATE MATERIALIZED VIEW expand `*` and qualify names in the definition they store; the parser reads `/*+ ... */` hints from comment tokens. The lexer counts its input by Unicode code point, while a Java `String` holds a supplementary character, such as an emoji, as two `char`s.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#47595 (introduced the bug in audit masking), apache#32743 (introduced the bug in view rewriting), apache#70338 (its review found the masking leak)
PR Type: Bugfix

Problem Summary:

After each supplementary character in a statement, these token positions point one `char` too early, so the text is cut at the wrong place. A credential after or inside such characters stays partly or fully in clear text in the audit log, CREATE VIEW fails with a syntax error, CREATE MATERIALIZED VIEW stores a broken query, and a hint after such a character is silently ignored. This PR converts the positions to `String` offsets, or reads the text through the token, at the five places that cut the original SQL. Tests are 89% of the diff.

### Details

- **Change map:** production code +33 -33 in 3 files; tests +240 in 5 files (FE unit tests and one regression suite); generated `.out` +16. (1) `BaseViewInfo.rewriteSql`, shared by audit masking, views and MTMV, converts its positions; the sync-MV builder drops its private copy of it, and `rewriteProjectsToUserDefineAlias` replaces the select list through it. (2) `NereidsParser.getHintMap` reads hints from the token text. (3) `NereidsParser.tryParseExplainPlan` converts the start of the EXPLAIN PLAN body.
- **Frequency:** Always, when a character above U+FFFF (an emoji, CJK Extension B) comes before or inside the rewritten part, for example a catalog COMMENT of emoji in front of `'s3.secret_key' = 'secret'`.
- **Root cause:** `CaseInsensitiveStream.fromString` keeps ANTLR's code-point stream for any input with a surrogate (before apache#67452 every input used `CharStreams.fromString`), so `Token.getStartIndex()` and `getStopIndex()` count code points. The five places in the Change map used them as `String` indexes.
- **Why existing tests missed it:** Unit tests: the masking (`EncryptSQLTest`, `RepositoryAuditEncryptionTest`), view and MTMV (`CreateViewTest`, `CreateMTMVCommandTest`) and hint tests (`LeanTokenModeTest`) use only BMP text, where both kinds of position coincide; `CaseInsensitiveStreamTest` uses an emoji but checks only the stream's own positions. Regression: `audit/test_external_catalog_credential_masking` and the `ddl_p0/create_view_nereids` suites have no supplementary characters, and the masking suite only checks that the whole credential is absent, which a leaked tail passes.
- **Fix:** Convert where the text is cut. `rewriteSql` converts each position with `String.offsetByCodePoints`, walking the sorted positions once so the cost stays linear; `rewriteProjectsToUserDefineAlias` reads expressions through the token's input stream; `getHintMap` reads the hint from the token text, as `removeCommentAndTrimBlank` does. Changing the stream to report `String` positions was rejected: apache#67452 made code-point positions part of `CaseInsensitiveStream`'s contract, and `CaseInsensitiveStreamTest` pins them.
- **Other paths and branches:** Fixed here: the 19 masking visitors of `LogicalPlanBuilderForEncryption` with every caller of `geneEncryptionSQL` (audit log, FE statement log, `_query_plan`) and the streaming insert job; the view, MTMV and sync-MV rewrites. Not affected: the other uses of token positions read text through the token's own stream (such as `getOriginSql`) or only compare positions with each other. branch-3.1, branch-4.0, branch-4.1 and branch-4.2 have the same code. Introduced by apache#32743 (views), apache#47595 (masking), apache#33699 (sync MV) and apache#33262 (EXPLAIN PLAN); hints since at least apache#39590.

### Testing

- New unit tests fail on master `f1bc8b3690c` and pass with the fix, run locally: `EncryptSQLTest#testEncryptionAfterSupplementaryCharacters`, through `MysqlConnectProcessor` to the audit event (emoji inside a credential, in a catalog COMMENT, in the first of two property clauses, in a user name before `PASSWORD(...)`; master left `secret`, `abc`, `123` and `123456` in clear text); `CreateViewTest#testCreateViewRewritesAfterSupplementaryCharacters` (master: `mismatched input '`e`'`); `CreateMTMVCommandTest#testCreateMTMVRewritesQuerySqlAfterSupplementaryCharacters` (master kept ``SELECT '😀😀' A`e` e`internal`...``); `SupplementaryCharacterOffsetTest` (master dropped the hint, cut the EXPLAIN PLAN body to `AN SELECT 100`, and rewrote the sync MV to `avg_stateg(k)))`).
- New regression suite `ddl_p0/create_view_nereids/test_create_view_supplementary_chars`: fails without the fix with the same syntax error and passes with it; the `.out` is generated by the script. Run locally on one FE and one BE, an older master FE build with the three changed classes swapped in.
- 27 existing FE UT classes around the change (467 tests) pass. Reverting each part of the fix alone, moving the end of a replaced span by one, or not advancing the position cursor each fails a new test; the off-by-one also fails existing ASCII cases.
- By hand on that cluster, without the fix: the secret stayed in `audit_log`, `/* 😀 */ select /*+ SET_VAR(query_timeout=1) */ sleep(3)` ran for 3 s, and an MTMV over `select '😀😀' as e, ...` was created but failed to refresh (`Nereids parse failed`). With the fix: `"s3.secret_key" = "*XXX")`, and the query times out after 1 s.

### Notes for Reviewers

- `getHintMap` loses its `sql` parameter, which nothing reads any more; its three callers are in `NereidsParser`.
- `rewriteProjectsToUserDefineAlias` replaces the select list through `rewriteSql` instead of `StringUtils.overlay`, so the conversion lives in one method.
- Stored definitions are not repaired: an MTMV created by an affected version from such a query keeps its broken definition and has to be recreated.

### Release note

Fix passwords and credentials left partly or fully unmasked in the audit log and the FE log when a supplementary character, such as an emoji, comes before or inside them. Also fix CREATE VIEW failing, CREATE MATERIALIZED VIEW storing a broken query, and hints such as SET_VAR being ignored after such a character; materialized views already created with a broken query must be recreated.

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: the fix, the tests, and this description
- Human review: The author reviewed part of the changes.
- Prompts: After the review bot on apache#70338 reported the masking leak, the author confirmed it is an existing engine defect shared by all masking paths and asked for a separate engine PR that fixes it, followed by a reply on apache#70338 with the link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
morningman pushed a commit to morningman/doris that referenced this pull request Nov 13, 2026
### Background

Connector plugins (Hudi, and the coming Lance connector) describe their columns with the SPI's `ConnectorType`, and fe-core's `ConnectorColumnConverter` turns each into a Doris type. Connectors put a temporal column's fractional-second scale in the type's precision field: `("DATETIMEV2", 3, 0)` is `datetime(3)`, and Hudi's `HudiTypeMapping` sends `("TIMEV2", 3, 0)` for an Avro `time-millis` column and `("TIMEV2", 6, 0)` for `time-micros`.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#62183 (introduced the converter without a TIMEV2 case), apache#64304 (moved Hudi onto it)
PR Type: Bugfix

Problem Summary:

Since Hudi became a connector plugin, its `time-millis` and `time-micros` columns show as `time` instead of `time(3)` and `time(6)`, which is what 4.x shows and what `HudiTypeMapping` asks for: the converter drops the scale and builds a scale-0 TIMEV2. The Lance connector, ported in a follow-up, maps Arrow `time32(ms)` and `time64` the same way and would lose its scale too. The converter now reads the TIMEV2 scale from the precision field, as it already does for DATETIMEV2.

### Details

- **Frequency:** Always, for every connector column whose `ConnectorType` is TIMEV2 with a scale; today that is every Hudi `time-millis` / `time-micros` column on master. Found while porting the Lance connector, by reading the converter, and confirmed by the unit test below.
- **Root cause:** `ConnectorColumnConverter.convertScalarType` reads the precision field for CHAR, VARCHAR, VARBINARY, the decimals, DATETIMEV2 and TIMESTAMPTZ, but has no TIMEV2 case, so TIMEV2 falls through to `ScalarType.createType("TIMEV2")`, which returns the scale-0 `ScalarType.TIMEV2`.
- **Why existing tests missed it:** Unit tests: `ConnectorColumnConverterTest` checks decimals, CHAR/VARCHAR lengths and the TIMESTAMPTZ extra info but has no TIMEV2 case, and `HudiTypeMappingTest` asserts the `("TIMEV2", 3, 0)` the plugin emits but stops before fe-core converts it. Regression: no suite under `regression-test/suites` reads a Hudi time column.
- **Fix:** The TIMEV2 case uses the precision field as the scale when one is given; without one the type stays the scale-0 TIMEV2 it has always been. The fix sits in the converter because every connector goes through it. The reverse direction, `toConnectorType`, is unchanged: only the read-only Hudi and Lance connectors produce TIMEV2, and the JDBC connector maps TIME to STRING.
- **Other paths and branches:** `toConnectorType` is not affected for reads (see Fix). branch-4.1 and branch-4.2 are not affected: they have no `ConnectorColumnConverter`, and their fe-core Hudi mapping (`HudiUtils`) builds `createTimeV2Type(3)` / `createTimeV2Type(6)` directly. The bug is on master since apache#64304.

### Testing

- `ConnectorColumnConverterTest#timeScaleRidesInThePrecisionSlot` (new): fails before the fix with `expected: <timev2(3)> but was: <timev2(0)>` and passes after; it also pins that a TIMEV2 without a precision stays scale 0. Run locally on master `f5c993f2c94` with apache#70338 applied, which touches neither file.
- `ConnectorColumnConverterTest` (35 tests) and `HudiTypeMappingTest` (17 tests): pass locally on the same tree.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: code, tests, this description
- Human review: not stated
- Prompts: The author asked to port the branch-4.2 Lance FE read path to master as a connector plugin; this fix came out of that port, and the author's rule for such finds is to fix them in their own PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@morningman
morningman force-pushed the b2m-t38c-connector-sensitive-keys branch from ae2cebe to b1fcc69 Compare November 13, 2026 08:39
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 43 changed paths at b1fcc6942705e80c80a7ae650b0006462d34df69 by static inspection, including related callers, consumers, tests and existing threads. No new inline findings.

Existing P1 4496273297 still applies to this head: the new ALTER visitor supplies code-point token offsets to BaseViewInfo.rewriteSql, which slices UTF-16. Supplementary characters can leave a credential suffix in audit/statement text. The separate fix referenced in the reply is absent from this source. This finding is carried by ID without another inline comment.

For existing P2 4492689479, both loading paths now catch the original ServiceConfigurationError. The directory path also catches LinkageError and discards its loader. A malformed shared test/embedded classpath can still produce a subsequent NoClassDefFoundError after metastore static initialization is poisoned; the builtin catch omits LinkageError. This remains within that existing thread's startup-failure mechanism. Production connector implementations are loaded from separate directories; their fe-core classpath dependencies are test-scoped. No duplicate inline finding is added.

The inspected control flow is:

Env startup -> provider securitySurface -> register exact keys / namespaces
                                         -> DatasourcePrintableMap
                                            -> SHOW CREATE / SHOW / catalogs()
ALTER command -> NeedAuditEncryption -> encryption visitor -> same masking policy
               -> BaseViewInfo.rewriteSql -> audit / FE statement logs

Critical checkpoint conclusions:

  1. Goal and proof: declarations reach catalog renderers and ordinary ALTER audit masking. Added unit and regression cases exercise these paths; the known Unicode case remains uncovered by this PR's additions.
  2. Scope and reuse: changes stay focused on masking metadata, startup registration and tests/docs, reusing annotations and the existing security surface.
  3. Concurrency and locks: production registration finishes before image loading/catalog access; the manager is published through a volatile reference. Immutable declarations and copy-on-write prefix iteration add no query-time writer or lock-order change. No new remote work under metadata locks was found.
  4. Lifecycle/static initialization: defining-classloader metastore discovery, publication order and rejected-directory disposal were traced. The original nested-loader failure and the classpath residual have the dispositions above.
  5. Configuration: no new configuration item or dynamic-update contract.
  6. Compatibility: connector API and its pinned test advance from 30.0 to 31.0, with the changed public signature baseline updated. Existing endpoint/driver-url declarations survive secret builders. No storage/wire format changes.
  7. Thrift/protobuf: no changed definitions or field IDs.
  8. Parallel paths: CREATE/ALTER audit, statement logging, SHOW renderings and catalog property consumers share the policy. JDBC/ES aliases, metastore flavor unions and ADBC/Trino/Fluss namespaces were checked; the legacy masking floor is preserved.
  9. Special conditions: prefix matching is case-insensitive, searches only the suffix for markers and supports an entire namespace. Keytab-path declassification matches the stated behavior. Missing-plugin and raw SDK/URL-embedded credential limits are explicit existing limits.
  10. Test coverage: inspected positive/negative matching, immutable builders, provider declarations, version gates, isolation, nested-loader faults, SHOW and ASCII ALTER audit cases. No builds, tests, benchmarks or runtime validation were executed under this static-only contract.
  11. Test results: no generated expected-output files change; no executed-result claim is made.
  12. Observability: skip warnings retain provider/directory and cause; no additional masking metric is needed.
  13. Persistence/failover: no EditLog, transaction or persistence-format changes; startup registration also covers restored catalog properties.
  14. Data writes/atomicity: no new data-write or commit behavior; data visibility, partition versions and delete bitmaps are unaffected.
  15. FE/BE propagation: no new cross-process variables or transmission sites.
  16. Performance/memory: reflection and unions occur at startup; rendering scans a small registered prefix list. No demonstrated performance or memory-ownership defect was found.
  17. Other issues and user focus: no additional focus was supplied. No further distinct verified issue or unreviewed changed scope remains after the reviewer rechecks. The known P1 remains blocking.

Coverage limits: static reasoning only; runtime behavior and the author's reported test runs were not independently executed. These limits are separate from review convergence.

Review converged after 2 round(s).

Existing P0/P1 findings confirmed for this head: #70338 (comment)

morningman added a commit that referenced this pull request Nov 16, 2026
### Background

Connector plugins (Hudi, and the coming Lance connector) describe their
columns with the SPI's `ConnectorType`, and fe-core's
`ConnectorColumnConverter` turns each into a Doris type. Connectors put
a temporal column's fractional-second scale in the type's precision
field: `("DATETIMEV2", 3, 0)` is `datetime(3)`, and Hudi's
`HudiTypeMapping` sends `("TIMEV2", 3, 0)` for an Avro `time-millis`
column and `("TIMEV2", 6, 0)` for `time-micros`.

### What problem does this PR solve?

Issue Number: None
Related PR: #62183 (introduced the converter without a TIMEV2 case),
#64304 (moved Hudi onto it)
PR Type: Bugfix

Problem Summary:

Since Hudi became a connector plugin, its `time-millis` and
`time-micros` columns show as `time` instead of `time(3)` and `time(6)`,
which is what 4.x shows and what `HudiTypeMapping` asks for: the
converter drops the scale and builds a scale-0 TIMEV2. The Lance
connector, ported in a follow-up, maps Arrow `time32(ms)` and `time64`
the same way and would lose its scale too. The converter now reads the
TIMEV2 scale from the precision field, as it already does for
DATETIMEV2.

### Details

- **Frequency:** Always, for every connector column whose
`ConnectorType` is TIMEV2 with a scale; today that is every Hudi
`time-millis` / `time-micros` column on master. Found while porting the
Lance connector, by reading the converter, and confirmed by the unit
test below.
- **Root cause:** `ConnectorColumnConverter.convertScalarType` reads the
precision field for CHAR, VARCHAR, VARBINARY, the decimals, DATETIMEV2
and TIMESTAMPTZ, but has no TIMEV2 case, so TIMEV2 falls through to
`ScalarType.createType("TIMEV2")`, which returns the scale-0
`ScalarType.TIMEV2`.
- **Why existing tests missed it:** Unit tests:
`ConnectorColumnConverterTest` checks decimals, CHAR/VARCHAR lengths and
the TIMESTAMPTZ extra info but has no TIMEV2 case, and
`HudiTypeMappingTest` asserts the `("TIMEV2", 3, 0)` the plugin emits
but stops before fe-core converts it. Regression: no suite under
`regression-test/suites` reads a Hudi time column.
- **Fix:** The TIMEV2 case uses the precision field as the scale when
one is given; without one the type stays the scale-0 TIMEV2 it has
always been. The fix sits in the converter because every connector goes
through it. The reverse direction, `toConnectorType`, is unchanged: only
the read-only Hudi and Lance connectors produce TIMEV2, and the JDBC
connector maps TIME to STRING.
- **Other paths and branches:** `toConnectorType` is not affected for
reads (see Fix). branch-4.1 and branch-4.2 are not affected: they have
no `ConnectorColumnConverter`, and their fe-core Hudi mapping
(`HudiUtils`) builds `createTimeV2Type(3)` / `createTimeV2Type(6)`
directly. The bug is on master since #64304.

### Testing

- `ConnectorColumnConverterTest#timeScaleRidesInThePrecisionSlot` (new):
fails before the fix with `expected: <timev2(3)> but was: <timev2(0)>`
and passes after; it also pins that a TIMEV2 without a precision stays
scale 0. Run locally on master `f5c993f2c94` with #70338 applied, which
touches neither file.
- `ConnectorColumnConverterTest` (35 tests) and `HudiTypeMappingTest`
(17 tests): pass locally on the same tree.
morningman added a commit that referenced this pull request Nov 16, 2026
…characters (#70366)

### Background

Several FE features rewrite the text of a statement by cutting the
original SQL string at token positions from the parser. Audit masking
replaces credential properties and passwords with `*XXX` before the
statement reaches `fe.audit.log`, `__internal_schema.audit_log` and the
FE log; CREATE / ALTER VIEW and CREATE MATERIALIZED VIEW expand `*` and
qualify names in the definition they store; the parser reads `/*+ ...
*/` hints from comment tokens. The lexer counts its input by Unicode
code point, while a Java `String` holds a supplementary character, such
as an emoji, as two `char`s.

### What problem does this PR solve?

Issue Number: None
Related PR: #47595 (introduced the bug in audit masking), #32743
(introduced the bug in view rewriting), #70338 (its review found the
masking leak)
PR Type: Bugfix

Problem Summary:

After each supplementary character in a statement, these token positions
point one `char` too early, so the text is cut at the wrong place. A
credential after or inside such characters stays partly or fully in
clear text in the audit log, CREATE VIEW fails with a syntax error,
CREATE MATERIALIZED VIEW stores a broken query, and a hint after such a
character is silently ignored. This PR converts the positions to
`String` offsets, or reads the text through the token, at the five
places that cut the original SQL. Tests are 89% of the diff.

For example, the eight emoji in this catalog's comment move both ends of
the span that masking replaces eight characters to the left, so the
audit log loses `ERTIES (` and keeps `"secret"` in clear text:

```sql
CREATE CATALOG hive COMMENT '😀😀😀😀😀😀😀😀'
PROPERTIES ("type" = "hms", "hive.metastore.uris" = "thrift://hms:9083", "s3.secret_key" = "secret");
```

| | Line 2 in the audit log (line 1 is unchanged) |
|---|---|
| Without the fix | `PROP"type" = "hms", "hive.metastore.uris" =
"thrift://hms:9083", "s3.secret_key" = "*XXX""secret")` |
| With the fix | `PROPERTIES ("type" = "hms", "hive.metastore.uris" =
"thrift://hms:9083", "s3.secret_key" = "*XXX")` |

### Details

- **Change map:** production code +33 -33 in 3 files; tests +240 in 5
files (FE unit tests and one regression suite); generated `.out` +16.
(1) `BaseViewInfo.rewriteSql`, shared by audit masking, views and MTMV,
converts its positions; the sync-MV builder drops its private copy of
it, and `rewriteProjectsToUserDefineAlias` replaces the select list
through it. (2) `NereidsParser.getHintMap` reads hints from the token
text. (3) `NereidsParser.tryParseExplainPlan` converts the start of the
EXPLAIN PLAN body.
- **Frequency:** Always, when a character above U+FFFF (an emoji, CJK
Extension B) comes before or inside the rewritten part, as in the
example above.
- **Root cause:** `CaseInsensitiveStream.fromString` keeps ANTLR's
code-point stream for any input with a surrogate (before #67452 every
input used `CharStreams.fromString`), so `Token.getStartIndex()` and
`getStopIndex()` count code points. The five places in the Change map
used them as `String` indexes.
- **Why existing tests missed it:** Unit tests: the masking
(`EncryptSQLTest`, `RepositoryAuditEncryptionTest`), view and MTMV
(`CreateViewTest`, `CreateMTMVCommandTest`) and hint tests
(`LeanTokenModeTest`) use only BMP text, where both kinds of position
coincide; `CaseInsensitiveStreamTest` uses an emoji but checks only the
stream's own positions. Regression:
`audit/test_external_catalog_credential_masking` and the
`ddl_p0/create_view_nereids` suites have no supplementary characters,
and the masking suite only checks that the whole credential is absent,
which a leaked tail passes.
- **Fix:** Convert where the text is cut. `rewriteSql` converts each
position with `String.offsetByCodePoints`, walking the sorted positions
once so the cost stays linear; `rewriteProjectsToUserDefineAlias` reads
expressions through the token's input stream; `getHintMap` reads the
hint from the token text, as `removeCommentAndTrimBlank` does. Changing
the stream to report `String` positions was rejected: #67452 made
code-point positions part of `CaseInsensitiveStream`'s contract, and
`CaseInsensitiveStreamTest` pins them.
- **Other paths and branches:** Fixed here: the 19 masking visitors of
`LogicalPlanBuilderForEncryption` with every caller of
`geneEncryptionSQL` (audit log, FE statement log, `_query_plan`) and the
streaming insert job; the view, MTMV and sync-MV rewrites. Not affected:
the other uses of token positions read text through the token's own
stream (such as `getOriginSql`) or only compare positions with each
other. branch-3.1, branch-4.0, branch-4.1 and branch-4.2 have the same
code. Introduced by #32743 (views), #47595 (masking), #33699 (sync MV)
and #33262 (EXPLAIN PLAN); hints since at least #39590.

### Testing

- New unit tests fail on master `f1bc8b3690c` and pass with the fix, run
locally: `EncryptSQLTest#testEncryptionAfterSupplementaryCharacters`,
through `MysqlConnectProcessor` to the audit event (emoji inside a
credential, in a catalog COMMENT, in the first of two property clauses,
in a user name before `PASSWORD(...)`; master left `secret`, `abc`,
`123` and `123456` in clear text);
`CreateViewTest#testCreateViewRewritesAfterSupplementaryCharacters`
(master: `mismatched input '`e`'`);
`CreateMTMVCommandTest#testCreateMTMVRewritesQuerySqlAfterSupplementaryCharacters`
(master kept ``SELECT '😀😀' A`e` e`internal`...``);
`SupplementaryCharacterOffsetTest` (master dropped the hint, cut the
EXPLAIN PLAN body to `AN SELECT 100`, and rewrote the sync MV to
`avg_stateg(k)))`).
- New regression suite
`ddl_p0/create_view_nereids/test_create_view_supplementary_chars`: fails
without the fix with the same syntax error and passes with it; the
`.out` is generated by the script. Run locally on one FE and one BE, an
older master FE build with the three changed classes swapped in.
- 27 existing FE UT classes around the change (467 tests) pass.
Reverting each part of the fix alone, moving the end of a replaced span
by one, or not advancing the position cursor each fails a new test; the
off-by-one also fails existing ASCII cases.
- By hand on that cluster, without the fix: the secret stayed in
`audit_log`, `/* 😀 */ select /*+ SET_VAR(query_timeout=1) */ sleep(3)`
ran for 3 s, and an MTMV over `select '😀😀' as e, ...` was created but
failed to refresh (`Nereids parse failed`). With the fix:
`"s3.secret_key" = "*XXX")`, and the query times out after 1 s.

### Notes for Reviewers

- `getHintMap` loses its `sql` parameter, which nothing reads any more;
its three callers are in `NereidsParser`.
- `rewriteProjectsToUserDefineAlias` replaces the select list through
`rewriteSql` instead of `StringUtils.overlay`, so the conversion lives
in one method.
- Stored definitions are not repaired: an MTMV created by an affected
version from such a query keeps its broken definition and has to be
recreated.
morningman added a commit that referenced this pull request Nov 16, 2026
…port at its default port (#70364)

### Background

When an operator configures `security_checker_class_name`, CREATE
CATALOG and ALTER CATALOG ask that checker whether the FE may connect to
each endpoint the catalog's connector declares through
`ConnectorSecuritySurface.ofSsrfEndpoints`, and refuse an intranet
target (#69588). `PluginDrivenExternalCatalog.parseEndpoint` turns each
declared value into the host and port to probe: `scheme://host:port`, or
a bare `host:port`. A value it cannot turn into a concrete host and port
is not probed.

### What problem does this PR solve?

Issue Number: None
Related PR: #69588 (added the gate)
PR Type: Bugfix

Problem Summary:

An http or https endpoint written without a port, such as
`http://10.0.0.1/ns` or `https://10.0.0.1`, passes the gate unprobed,
although an HTTP client connects to port 80 or 443 for such a URI. Today
the Hive and MongoDB connectors declare endpoints, as thrift and
`mongodb://` URIs, so no connector on master reaches this yet; the first
http endpoint will be the Lance REST catalog's `lance.rest.uri`, and an
https service URL usually names no port. The gate now probes such an
endpoint at its scheme's default port.

### Details

- **Frequency:** Not observed; found by code review while declaring the
Lance REST catalog's endpoint. Triggers when an SSRF checker is
configured and a declared endpoint is an http or https URL without a
port.
- **Root cause:** `parseEndpoint` returned null for a `scheme://` value
whose `URI.getPort()` is -1, and `checkEndpointsAgainstSsrf` skips a
null as nothing to probe.
- **Why existing tests missed it:** Unit tests:
`PluginDrivenExternalCatalogSsrfGateTest` covers thrift URIs, bare
`host:port`, IPv6 and URI suffixes, and asserts that a thrift URI
without a port is not probed; no case used an http or https URL.
Regression: no suite reaches the gate, since it needs the operator's
security checker SDK, which CI does not have; the unit tests use a fake.
- **Fix:** For an http or https URI that names no port, `parseEndpoint`
takes 80 or 443, the ports RFC 9110 assigns to the scheme; an explicit
port still wins. A URI of another scheme without a port is still not
probed: its default port, if it has one, belongs to the client, and the
engine knows no connector's client. The fix is in the engine because the
port follows from the scheme alone.
- **Other paths and branches:** CREATE (`checkWhenCreating`) and both
ALTER paths go through `parseEndpoint`, so all are fixed. branch-4.1 and
branch-4.2 have the same rule in `HMSExternalCatalog.parseEndpoint`, but
gate only Hive metastore thrift URIs, so they are not affected.
Introduced by #69588.

### Testing

Local (macOS), on master `f1bc8b3690c` plus this change:

- `PluginDrivenExternalCatalogSsrfGateTest`: the new
`parseEndpointUsesTheDefaultPortOfAWebScheme` (seven URIs: with and
without a path, an upper-case scheme, user-info, query and fragment,
IPv6, an explicit port) and
`alterRejectsAnIntranetWebEndpointWithoutAPort` (an http and an https
endpoint through the real ALTER validation, with `10.0.0.1` flagged as
intranet). On the code before this change 8 of them fail: the six URIs
without a port parse to null, and both ALTER cases accept the endpoint
("Expected DdlException to be thrown, but nothing was thrown"). With it,
all 44 cases of the class pass, as do the 18 of
`PluginDrivenExternalCatalogDriverUrlGateTest`; fe-core checkstyle
passes.

### Notes for Reviewers

- The Javadoc of `ConnectorSecuritySurface.ofSsrfEndpoints` still
describes the value as a `host:port`, optionally scheme-prefixed. It is
left alone here because #70338, still open, edits that file.
morningman and others added 3 commits November 16, 2026 21:11
…roperties to mask

### Background

SHOW CREATE CATALOG, and the other places that print catalog properties, replace a value with `*XXX` when `DatasourcePrintableMap.isSensitiveKey` matches its key. The keys come from a list hard-coded in fe-core and from the keys that filesystem plugins report at startup through `FileSystemProvider.sensitivePropertyKeys()`. A connector plugin tells the engine about its security-relevant properties through `ConnectorProvider#securitySurface()`, which today lists only the SSRF endpoint keys and the jdbc `driver_url` keys that the CREATE and ALTER CATALOG gates check.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69588 (added the connector security surface this extends), apache#69575 (master loads the Lance catalogs that 4.1.4 persisted), apache#65730 (branch-4.1's Lance catalog, whose two REST secrets 4.x masks in fe-core)
PR Type: Enhancement

Problem Summary:

A connector plugin parses its own catalog properties, but their secret values are masked only if fe-core hard-codes the key names, which the connector framework rules out. The Lance connector being ported from branch-4.x needs this for `lance.rest.bearer-token` and `lance.rest.api-key`: 4.1.4 masks them, and without a way to declare them master would print them in clear. `ConnectorSecuritySurface` can now declare secret keys, and the engine adds an admitted provider's keys to its masking list at FE startup. Nothing changes after an upgrade until a connector declares keys, and none does in this PR.

### Details

- **Before and after:** Before, a property that only a connector knows, such as `lance.rest.bearer-token`, showed its value in SHOW CREATE CATALOG. After, a key the connector declares with `ConnectorSecuritySurface.ofSensitiveKeys(...)`, or with `.withSensitiveKeys(...)` on top of its other declarations, shows `*XXX`, matched case-insensitively like the engine's own list; undeclared keys show as before.
- **Configs and defaults:** None. The default surface declares no secret.
- **Impact on existing users:** None for SQL, metadata, or configs. As for every change to the connector SPI surface, the plugin API major moves from 31.0 to 32.0, so a third-party connector plugin built against 31.x has to be rebuilt; the in-tree plugins are built with the FE.

### Testing

All runs were local (macOS), on master `59c49656240` plus this change:

- `mvn -f fe/pom.xml -pl :fe-connector-spi -am test`: the module's 156 tests pass. They include the new `ConnectorSecuritySurfaceTest`, which covers combining secrets with the SSRF and driver_url declarations, appending, and that no existing surface, `EMPTY` included, is changed. They also include `ConnectorPluginSurfaceTest` with the refreshed baseline and 31.0.
- fe-core, 12 classes and 157 tests, all passing. `ConnectorPluginManagerTest` has three new cases:
  - an admitted provider's key is masked, case-insensitively, in the rendering SHOW CREATE CATALOG uses, and undeclared keys stay visible;
  - a refused provider adds nothing;
  - the `registerProvider` test seam registers keys too.
- The other eleven classes register providers or check the security surface, such as `PluginDrivenExternalCatalog{SsrfGate,DriverUrlGate}Test`, `CatalogFactoryPluginRoutingTest` and `DatasourcePrintableMapTest`.
- Six mutations, each caught by the test written for it:
  - not registering;
  - registering before the admission checks;
  - skipping the test seam;
  - three in `withSensitiveKeys`: dropping the SSRF keys, replacing instead of appending, and returning a mutable list.

No connector declares keys yet, so nothing changes end to end. The Lance connector, ported in follow-up PRs, will be the first to declare them.

### Notes for Reviewers

- The keys are read when a provider is registered and added only after the admission checks in `ConnectorPluginManager#registerDiscovered` pass, so a refused provider adds nothing. Connector plugins load only at FE startup, before any catalog property is shown, as filesystem plugins do. `registerProvider`, the seam tests use to stand in for a plugin, registers the keys too, like `FileSystemPluginManager#registerProvider`.
- A catalog whose connector plugin is not loaded (a degraded catalog) keeps only the engine's own list.
- Why the surface rather than a provider method like `FileSystemProvider.sensitivePropertyKeys()`: the connector SPI already gathers its security-relevant property keys on the surface, and a connector with both an endpoint and a secret declares them on one object.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: the code, the tests, and this description
- Human review: not stated
- Prompts: The author asked for a generic connector SPI to mask a connector's secret catalog properties instead of adding the Lance keys to fe-core's list, approved the `ConnectorSecuritySurface` design, and asked for it as a separate PR ahead of the Lance connector port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…udit text

### Background

SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit text of CREATE CATALOG replace a property value with `*XXX` when `DatasourcePrintableMap.isSensitiveKey` matches its key. Until now the keys came from a list in fe-core and from the keys filesystem plugins report at startup. A connector plugin marks its secret properties `sensitive = true`, but that mark only masks the connector's own log lines. A connector tells the engine about its security-relevant properties through `ConnectorProvider#securitySurface()`.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69588 (added the connector security surface this extends), apache#45164 (introduced the unmasked ALTER CATALOG audit text), apache#27115 (masked it for the statement apache#45164 replaced), apache#59502 (added `iceberg.jdbc.password`), apache#54523 (added the Paimon REST DLF keys), apache#65730 (branch-4.1's Lance catalog, whose REST secrets the Lance port will declare on this surface)
PR Type: Bugfix

Problem Summary:

Catalog secrets that only a connector knows are on no masking list, so SHOW CREATE CATALOG and the other renderings print them in clear: `iceberg.jdbc.password`, `paimon.jdbc.password`, the Paimon REST DLF key pair, `mc.access_key`, and credentials passed through under `trino.` or `adbc.`. Separately, `ALTER CATALOG ... SET PROPERTIES` is written to the audit log unmasked, whatever the key. Each connector now declares its secrets on its security surface, taken from its `sensitive` marks, and the surface can also declare a namespace of secret keys for options passed through unread; Fluss's lake options were hard-coded in fe-core for that reason. The engine masks the union, with fe-core's own list kept as the floor. ALTER CATALOG is masked like CREATE CATALOG.

### Details

- **Change map:** 42 files, +1035 −76 over two commits: production code +376 −48, tests +645 −24, the connector README +13 −3, the plugin API version in `fe/fe-connector/pom.xml`. The first commit adds secret keys to the surface (reviewed alone before). The second does the rest:
  1. Key namespaces on the SPI: `withSensitiveKeyPrefix` and `SensitiveKeyPrefix.covers`, which holds the matching rule; a collection form of `withSensitiveKeys` (`ConnectorSecuritySurface`, +130).
  2. Engine: `isSensitiveKey` matches registered namespaces, the Fluss rule becomes one of them; `ConnectorPluginManager` registers them and reads every surface before it publishes a provider (+70).
  3. Connectors: every provider declares its secrets; `MetaStoreProviders.sensitivePropertyKeys()` gives the union of a plugin's metastore backends; `IcebergJdbcMetaStoreProvider` reports its keys like its siblings; the keytab paths lose their `sensitive` mark (+140).
  4. ALTER CATALOG audit masking (+16).
- **Frequency:** Always. Create or alter a catalog that sets one of the keys above, then run SHOW CREATE CATALOG, SHOW CATALOG `<name>` or `select * from catalogs()`, or read the audit log after an ALTER CATALOG ... SET PROPERTIES.
- **Root cause:** `isSensitiveKey` consulted only fe-core's list and the filesystem providers' keys. A connector's `sensitive` marks fed only `toMaskedString`; `MetaStoreProvider.sensitivePropertyKeys()` existed for this, but nothing called it, and the Iceberg JDBC backend did not implement it. For ALTER CATALOG, `AlterCatalogPropertiesCommand` does not implement `NeedAuditEncryption` and `LogicalPlanBuilderForEncryption` has no `visitAlterCatalogProperties`, so `AuditLogHelper` records the original text; the statement it replaced in apache#45164 was masked by apache#27115.
- **Why existing tests missed it:** Unit tests: `DatasourcePrintableMapTest` checks fe-core's own keys, the provider surface tests checked only the SSRF and driver_url keys, and `EncryptSQLTest` and `RepositoryAuditEncryptionTest` cover CREATE CATALOG, repositories and resources but not ALTER CATALOG. Regression: `audit/test_external_catalog_credential_masking` creates only Iceberg REST catalogs, whose keys are on fe-core's list, and never alters one.
- **Fix:** The engine must not name a connector's properties, so the connectors declare their secrets, from the same `sensitive` marks that mask their logs; jdbc and es declare both spellings their legacy prefix allows, and trino and adbc declare the namespace they pass through, a key there being a secret when its rest names `key`, `secret`, `password`, `token` or `credential` (also `authorization` for ADBC). fe-core keeps the connector keys it lists, so a catalog whose plugin is not loaded masks what it masked before. A provider whose `securitySurface()` throws at startup is skipped, as a filesystem provider is, instead of stopping FE.
- **Other paths and branches:** Fixed here: every rendering above and FE's statement logging, which share `isSensitiveKey` and `NeedAuditEncryption`. Not covered, since a key name cannot express them safely: raw Iceberg SDK keys copied through (`token`, `credential`), Paimon options re-keyed from `paimon.*`, Hadoop `fs.*` keys, and secrets inside a URL. The ALTER CATALOG gap is on branch-4.0, 4.1 and 4.2 (from apache#45164); branch-3.1 still uses the masked statement. branch-4.1 and 4.2 also print the connector keys above in clear; without the connector SPI their fix differs and is tracked separately.
- **Behavior change:** The namespaces also mask non-secrets whose names carry those words, such as `trino.hive.s3.aws-access-key` or a keystore path. Keytab paths, which no release masks, are no longer masked in connector log lines: marked `sensitive`, `hadoop.kerberos.keytab` would be masked in every storage vault and resource.

### Testing

All runs were local (macOS), on master `f5c993f2c94` plus this change:

- Regression `audit/test_external_catalog_credential_masking`, extended with an Iceberg JDBC catalog: SHOW CREATE CATALOG masks `iceberg.jdbc.password`, and the audit texts of its CREATE CATALOG and of an ALTER CATALOG that sets the password and `s3.secret_key` hold only `*XXX`. Against the local cluster's FE without this change it fails at the first new check, SHOW CREATE CATALOG printing the password; with a FE built from this change it passes.
- fe-connector-spi: all 160 tests, with the refreshed surface baseline (+4 lines); the metastore modules: all tests; ten connector modules: 29 classes, 511 tests, including each connector's surface test, which checks real key names.
- fe-core: 26 classes, 341 tests, including `ConnectorPluginManagerTest`, `DatasourcePrintableMapTest`, `EncryptSQLTest` and the audit and plugin gate tests; 14 of them again after the last change to registration.
- Twenty mutations, each caught by its test: for example, reverting either half of the ALTER CATALOG fix (`EncryptSQLTest` sees the statement in clear), matching a marker anywhere in the key, an empty metastore union, or re-marking a keytab.

### Notes for Reviewers

- `SensitiveKeyPrefix.covers` lives on the SPI so that a connector's test can check its declaration against real key names and the engine applies the same rule; the Fluss floor in fe-core uses it too.
- The entries both fe-core and a connector declare are deliberate: the floor is frozen, as the comment in `DatasourcePrintableMap` says.
- The plugin API stays at 31.0, the first commit's bump. Hudi declares nothing: it owns no catalog properties.

### Release note

SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit log now mask `iceberg.jdbc.password`, `paimon.jdbc.password`, `paimon.rest.dlf.access-key-id`, `paimon.rest.dlf.access-key-secret`, `mc.access_key`, and the `trino.*` and `adbc.*` properties whose names carry `key`, `secret`, `password`, `token` or `credential` (`authorization` too for `adbc.*`). ALTER CATALOG ... SET PROPERTIES statements are masked in the audit log like CREATE CATALOG.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: the code, the tests, and this description
- Human review: not stated
- Prompts: The author first asked for a generic connector SPI to mask a connector's secret catalog properties instead of adding the Lance keys to fe-core's list. Then, in this PR: let the surface express prefix rules such as Fluss's, mask the union of the engine's and the connectors' keys, wire every connector, keep fe-core's existing entries as the floor, mask whatever is marked `sensitive` but stop marking keytab paths, and fix the unmasked ALTER CATALOG audit text on master only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n its own ServiceLoader

### Background

FE reads every connector provider's security surface at startup, before it publishes the provider, and skips a provider whose surface throws, so that one broken plugin directory does not stop FE. The Hive, Iceberg and Paimon providers build theirs from their metastore backends' secret keys, which `MetaStoreProviders` collects in a static initializer that runs a `ServiceLoader` over the backends the plugin's jars register.

### What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

A `ServiceLoader` that cannot load a backend its descriptors name throws `ServiceConfigurationError`, an `Error` that is neither a `RuntimeException` nor a `LinkageError`, and a static initializer rethrows an `Error` as it is. The plugin directory path caught only those two and the classpath path only `RuntimeException`, so a descriptor naming a backend that is not there, such as one in a jar an earlier release left in a connector's `lib/`, stopped FE at startup; before the previous commit, which reads the surfaces at startup, it failed only that connector's catalogs. Both paths now skip the provider on a `ServiceConfigurationError` as well, and the plugins next to it still load.

### Release note

None

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@morningman
morningman force-pushed the b2m-t38c-connector-sensitive-keys branch from b1fcc69 to 9abe6cc Compare November 16, 2026 13:23
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43875540650

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

…udit text

### Background

SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit text of CREATE CATALOG replace a property value with `*XXX` when `DatasourcePrintableMap.isSensitiveKey` matches its key. Until now the keys came from a list in fe-core and from the keys filesystem plugins report at startup. A connector plugin marks its secret properties `sensitive = true`, but that mark only masks the connector's own log lines. A connector tells the engine about its security-relevant properties through `ConnectorProvider#securitySurface()`.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69588 (added the connector security surface this extends), apache#45164 (introduced the unmasked ALTER CATALOG audit text), apache#27115 (masked it for the statement apache#45164 replaced), apache#59502 (added `iceberg.jdbc.password`), apache#54523 (added the Paimon REST DLF keys), apache#65730 (branch-4.1's Lance catalog, whose REST secrets the Lance port will declare on this surface), apache#70366 (fixed the shared Unicode SQL rewrite used by this PR's ALTER masking), apache#70327 (added MongoDB credentials to the engine masking floor)
PR Type: Bugfix

Problem Summary:

Catalog secrets that only a connector knows are on no masking list, so SHOW CREATE CATALOG and the other renderings print them in clear: `iceberg.jdbc.password`, `paimon.jdbc.password`, the Paimon REST DLF key pair, `mc.access_key`, and credentials passed through under `trino.` or `adbc.`. Separately, `ALTER CATALOG ... SET PROPERTIES` is written to the audit log unmasked, whatever the key. Each connector now declares its secrets on its security surface, taken from its `sensitive` marks, and the surface can also declare a namespace of secret keys for options passed through unread; Fluss's lake options were hard-coded in fe-core for that reason. The engine masks the union, with fe-core's own list kept as the floor. ALTER CATALOG is masked like CREATE CATALOG.

### Details

- **Change map:** 45 files, +1205 −81 over four commits: production code +384 −51, tests +807 −26, the connector README +13 −3, the plugin API version in `fe/fe-connector/pom.xml` (31.0 to 32.0). The first commit adds secret keys to the surface (reviewed alone before). The second does the rest, the third keeps a surface that fails in the connector's own `ServiceLoader` from stopping FE, and the fourth aligns the newly added MongoDB provider and tests ALTER masking with supplementary characters:
  1. Key namespaces on the SPI: `withSensitiveKeyPrefix` and `SensitiveKeyPrefix.covers`, which holds the matching rule; a collection form of `withSensitiveKeys` (`ConnectorSecuritySurface`).
  2. Engine: `isSensitiveKey` matches registered namespaces, the Fluss rule becomes one of them; `ConnectorPluginManager` registers them and reads every surface before it publishes a provider, skipping a provider whose surface throws.
  3. Connectors: every provider declares its secrets; `MetaStoreProviders.sensitivePropertyKeys()` gives the union of a plugin's metastore backends; `IcebergJdbcMetaStoreProvider` reports its keys like its siblings; the keytab paths lose their `sensitive` mark. MongoDB also declares its two annotated secrets, preserving its SSRF endpoint and the engine fallback.
  4. ALTER CATALOG audit masking (+16).
- **Frequency:** Always. Create or alter a catalog that sets one of the keys above, then run SHOW CREATE CATALOG, SHOW CATALOG `<name>` or `select * from catalogs()`, or read the audit log after an ALTER CATALOG ... SET PROPERTIES.
- **Root cause:** `isSensitiveKey` consulted only fe-core's list and the filesystem providers' keys. A connector's `sensitive` marks fed only `toMaskedString`; `MetaStoreProvider.sensitivePropertyKeys()` existed for this, but nothing called it, and the Iceberg JDBC backend did not implement it. For ALTER CATALOG, `AlterCatalogPropertiesCommand` does not implement `NeedAuditEncryption` and `LogicalPlanBuilderForEncryption` has no `visitAlterCatalogProperties`, so `AuditLogHelper` records the original text; the statement it replaced in apache#45164 was masked by apache#27115.
- **Why existing tests missed it:** Unit tests: `DatasourcePrintableMapTest` checks fe-core's own keys, the provider surface tests checked only the SSRF and driver_url keys, and `EncryptSQLTest` and `RepositoryAuditEncryptionTest` cover CREATE CATALOG, repositories and resources but not ALTER CATALOG. Regression: `audit/test_external_catalog_credential_masking` creates only Iceberg REST catalogs, whose keys are on fe-core's list, and never alters one.
- **Fix:** The engine must not name a connector's properties, so the connectors declare their secrets, from the same `sensitive` marks that mask their logs; jdbc and es declare both spellings their legacy prefix allows, and trino and adbc declare the namespace they pass through, a key there being a secret when its rest names `key`, `secret`, `password`, `token` or `credential` (also `authorization` for ADBC). fe-core keeps the connector keys it lists, so a catalog whose plugin is not loaded masks what it masked before. A provider whose `securitySurface()` throws at startup is skipped, as a filesystem provider is, instead of stopping FE; that includes the `ServiceConfigurationError` a `ServiceLoader` throws when a descriptor names a backend it cannot load, which is how the Hive, Iceberg and Paimon surfaces fail, since they read their metastore backends' keys through one.
- **Other paths and branches:** Fixed here: every rendering above and FE's statement logging, which share `isSensitiveKey` and `NeedAuditEncryption`. Not covered, since a key name cannot express them safely: raw Iceberg SDK keys copied through (`token`, `credential`), Paimon options re-keyed from `paimon.*`, Hadoop `fs.*` keys, and secrets inside otherwise unmarked URLs. The ALTER CATALOG gap is on branch-4.0, 4.1 and 4.2 (from apache#45164); branch-3.1 still uses the masked statement. branch-4.1 and 4.2 also print the connector keys above in clear; without the connector SPI their fix differs and is tracked separately.
- **Behavior change:** The namespaces also mask non-secrets whose names carry those words, such as `trino.hive.s3.aws-access-key` or a keystore path. Keytab paths, which no release masks, are no longer masked in connector log lines: marked `sensitive`, `hadoop.kerberos.keytab` would be masked in every storage vault and resource.

### Testing

The original local runs (macOS) used master `f5c993f2c94` plus this change:

- Regression `audit/test_external_catalog_credential_masking`, extended with an Iceberg JDBC catalog: SHOW CREATE CATALOG masks `iceberg.jdbc.password`, and the audit texts of its CREATE CATALOG and of an ALTER CATALOG that sets the password and `s3.secret_key` hold only `*XXX`. Against the local cluster's FE without this change it fails at the first new check, SHOW CREATE CATALOG printing the password; with a FE built from this change it passes.
- fe-connector-spi: all 160 tests, with the refreshed surface baseline (+4 lines); the metastore modules: all tests; ten connector modules: 29 classes, 511 tests, including each connector's surface test, which checks real key names.
- fe-core: 26 classes, 341 tests, including `ConnectorPluginManagerTest`, `DatasourcePrintableMapTest`, `EncryptSQLTest` and the audit and plugin gate tests; 14 of them again after the last change to registration, and 7 classes, 76 tests, after the third commit. Its two new `ConnectorPluginManagerTest` cases, a plugin directory whose `lib/` holds a jar with a service descriptor naming a missing metastore-style backend (the plugin next to it must still load) and the same descriptor on the classpath, fail without the third commit with that `ServiceConfigurationError` escaping `loadPlugins` and `loadBuiltins`, and pass with it.
- Twenty-two mutations, each caught by its test: for example, reverting either half of the ALTER CATALOG fix (`EncryptSQLTest` sees the statement in clear), matching a marker anywhere in the key, an empty metastore union, re-marking a keytab, or dropping `ServiceConfigurationError` from either startup catch (each fails only its own new case).

Rebased onto master `93c16ee2ca1` on 2026-11-16. Post-rebase local verification:

- `bash run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`, with `CUSTOM_MVN` appending `-Dcheckstyle.skip=false`: BUILD SUCCESS across all 90 reactor modules; all 141 selected tests passed across 22 classes, including the inherited supplementary-character audit cases. Checkstyle reported no violations.
- `git diff --check upstream-apache/master...HEAD`: passed. `git range-diff` confirms the replayed patches are unchanged apart from adapting the connector API version and its test to 32.0.
- Regression tests and a complete BE/FE build were not rerun locally after this rebase; `run buildall` is requested in CI.

Follow-up local verification on the rebased branch plus the MongoDB declaration and audit tests (2026-11-16):

- `run-fe-ut.sh --run 'org.apache.doris.connector.spi.*Test,org.apache.doris.connector.mongodb.*Test,ConnectorPluginManagerTest,DatasourcePrintableMapTest,EncryptSQLTest'`, with Checkstyle enabled: all 176 selected tests in 28 classes passed (SPI 93, MongoDB 35, fe-core 48), with no skipped tests; the 90-module reactor completed successfully.
- The strengthened `MongoProviderTest` fails without the provider declaration (expected the two MongoDB secret keys, received an empty set), and passes with it. A separate targeted revert confirms the assertion catches the missing declaration.
- The new ALTER CATALOG assertions cover supplementary characters within a secret, in a preceding SQL comment, and in a retained non-secret value, with two masked properties. `EncryptSQLTest#testEncryption` passes on the candidate and fails at the new ALTER assertion when only `BaseViewInfo.rewriteSql` is restored to its pre-apache#70366 implementation: the audit text has an extra trailing fragment.
- Full cluster regression and a standalone FE/BE build were not rerun for this follow-up.

### Notes for Reviewers

- `SensitiveKeyPrefix.covers` lives on the SPI so that a connector's test can check its declaration against real key names and the engine applies the same rule; the Fluss floor in fe-core uses it too.
- The entries both fe-core and a connector declare are deliberate: the floor is frozen, as the comment in `DatasourcePrintableMap` says. MongoDB was already masked by apache#70327; its added declaration makes the provider follow the same contract as the other connectors.
- The connector plugin API advances from 31.0 to 32.0 for the added security-surface methods. Third-party connector plugins built against 31.x must be rebuilt; the in-tree plugins are built with FE. Hudi declares nothing: it owns no catalog properties.
- The base includes apache#70366, so ALTER CATALOG masking uses the shared rewrite that converts parser code-point spans to Java string offsets, including supplementary characters.

### Release note

SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit log now mask `iceberg.jdbc.password`, `paimon.jdbc.password`, `paimon.rest.dlf.access-key-id`, `paimon.rest.dlf.access-key-secret`, `mc.access_key`, and the `trino.*` and `adbc.*` properties whose names carry `key`, `secret`, `password`, `token` or `credential` (`authorization` too for `adbc.*`). ALTER CATALOG ... SET PROPERTIES statements are masked in the audit log like CREATE CATALOG. Third-party connector plugins built against API 31.x must be rebuilt for 32.0.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5); OpenAI Codex
- Generated: Claude Code wrote the original code, tests, and description. Codex rebased the commits, adapted the connector API version and its test, added the MongoDB declaration and ALTER CATALOG supplementary-character tests, and updated this description.
- Human review: not stated
- Prompts: The author requested a generic connector surface, prefix rules, declarations for every connector, and masking of their union with the engine floor. They removed sensitive marks from keytab paths, added ALTER CATALOG masking, and fixed startup failure from a connector's ServiceLoader. After review, they chose a separate shared Unicode fix, then requested MongoDB alignment and supplementary-character audit coverage here.
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43880814316

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70338
commit: ab5d64aa529cf5d0a102cb2dfd56e2feba0fe592
base: 5e85976d4c58d5528cb49d3ef677d5e695c2a36d
reviewed_at: 2026-11-17T10:58+08:00
reviewer: morningman
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 1, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • F-01 (Minor, test-coverage; Regression: no) fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorSecuritySurface.java:173-179: add the public SensitiveKeyPrefix type to the frozen plugin API inventory so prefix(), markers() and covers(String) are checked. Current masking behavior and the API 32.0 bump are intact.
  • Review covered all 45 changed files and converged in round 1; builds and tests were not run during this read-only review.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the FE changes and the related startup, catalog rendering, and audit SQL masking paths. No additional blocking issues found. No local builds or tests were run during this review. Compile, FE UT, P0, external, and nonconcurrent CI checks passed; the performance check is failing and its cause has not been verified.

@morningman
morningman merged commit 0744a64 into apache:master Nov 17, 2026
36 of 38 checks passed
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Nov 17, 2026
### Background

The spatial PR extends the connector SPI. Upstream master independently added secret-property declarations to that SPI and advanced the connector API to the same major.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032, apache#70338 (upstream connector API change)
PR Type: Build/CI/Chore

Problem Summary:

Merge master 9b5352b into the spatial branch while retaining both SPI surfaces. Resolve the API-32 version collision by advancing the merged spatial contract to API 33 and checking that pre-spatial API-32 plugins are rejected.

### Details

- **Change map:** Merge upstream master; manually reconcile only the connector parent POM, its recorded-surface version test, and the engine plugin-version wiring test. Other changes retain the upstream merge result.
- **Affected:** Connector plugin API version declaration and its two version tests.
- **Developer and release impact:** In-tree connectors share API 33; third-party connector plugins need the spatial API contract. Preserve both the spatial and upstream secret-property surface baselines.
- **License and third-party:** No additional dependency or license change beyond the upstream merge.
- **Rollback:** Revert this merge while retaining its first parent if necessary.

### Testing

- The personal standard FE UT wrapper passed all 291 tests in 39 classes: the complete connector SPI suite, IcebergWritePlanProviderTest and PluginApiVersionWiringTest; zero failures, errors or skips.
- The personal standard BE UT wrapper compiled the merged ASAN sources and passed all 292 tests in eight Iceberg, table-reader and cache suites; zero failures or skips. Local UT GLIBC_COMPATIBILITY=OFF; task-owned dependency adapter restored.
- The initial BE run had three fixture-setup failures from inaccessible shared temporary directories. The same selection passed with a private TMPDIR, without source changes or changing those directories.
- Clang-format16 checked 5145 files; build hygiene and full FE reactor CheckStyle passed. Code whitespace checks passed; generated regression golden formatting was preserved.
- Complete product rebuild, SQL regression, full unit-test matrix and a new full PR review were not run for this merge.

### Release note

None.

### Check List (For Author)

- Behavior changed: Yes, connector API advances to 33 to distinguish the merged spatial contract from upstream API 32.
- Does this need documentation: No
- Pre-submission status: Partially verified: targeted merge checks passed; complete community CI remains separate.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6).
- Generated: The merge conflict resolution, API-version test adjustments and this message.
- Human review: Pending author review.
Owen-CH-Leung added a commit to qube-rt/apache-doris that referenced this pull request Nov 17, 2026
…VARBINARY_COLUMN

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70338 (bumped the major to 32.0 for the catalog secret surface)
PR Type: Build/CI/Chore

Problem Summary: apache#70338 landed after this branch was written and also took the
plugin API major from 31.0 to 32.0, for its own SPI change. Rebasing left
SUPPORTS_VARBINARY_COLUMN sharing that major, and the merge conflict was in
ConnectorPluginSurfaceTest. fe/fe-connector/pom.xml says every SPI surface
change gets its own major, so this moves it to 33.0. It supersedes the
"31.0 to 32.0" in the previous commit's description.

### Details

- **Affected:** connector.plugin.api.version in fe/fe-connector/pom.xml and the
  matching assertion and comment in ConnectorPluginSurfaceTest.
- **Developer and release impact:** connector plugins built against API 32.x
  need a rebuild.
- **License and third-party:** None
- **Rollback:** revert with the previous commit.

### Testing

ConnectorPluginSurfaceTest passes locally in the apache/doris build-env
container.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus)
- Generated: this commit and its description
- Human review: pending
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants