Skip to content

[fix](catalog) Mask JDBC and REST catalog URI credentials - #70696

Open
OIiveirra wants to merge 5 commits into
apache:masterfrom
OIiveirra:fix/jira-security-320
Open

OIiveirra wants to merge 5 commits into
apache:masterfrom
OIiveirra:fix/jira-security-320

Conversation

@OIiveirra

Copy link
Copy Markdown
Collaborator

Background

Catalogs store JDBC and REST connection URIs, which can include credentials without a separate password property. Connector providers export sensitive property names and aliases at FE startup; catalog displays use the shared registry to mask their values.

What problem does this PR solve?

Issue Number: None
Related PR: #70338 (added connector-declared masking and metastore registration)
PR Type: Bugfix

Problem Summary:

Catalog readers with SELECT can still see credentials embedded in JDBC and REST URI properties because those properties are not declared sensitive. Mark the five URI fields sensitive so the existing registration chain covers their aliases, including the legacy JDBC prefix. Most of the change adds checks for aliases, preserved connection values, and SELECT-only display access.

Details

  • Frequency: Always when an affected URI contains credentials and a catalog reader displays its properties.
  • Root cause: The JDBC and Iceberg/Paimon JDBC/REST URI annotations omit sensitive=true, so their names never reach the provider's masking surface.
  • Why existing tests missed it: Property tests covered binding and standalone passwords; provider tests omitted URI keys, and the Iceberg dispatch test explicitly expected uri to remain unmasked. The existing audit regression checks standalone passwords and audit text, without SELECT-only URI alias checks.
  • Fix: Declare URI sensitivity in the property holders and reuse the existing alias expansion, metastore union, and provider registration. Mask complete URI values because JDBC and REST support different credential encodings.
  • Other paths and branches: Covers SHOW CREATE CATALOG, SHOW CATALOG <name>, and catalogs(), plus annotation-based property masking. Both jdbc_url and jdbc.jdbc_url are covered. Only master was verified; release-branch applicability and the earliest introducing changes remain unconfirmed.
  • Behavior change: These URI values are masked even without embedded credentials. The shared uri alias is masked wherever the common registry is used. Stored connection values and ordinary properties retain their behavior; keytab paths follow the upstream visibility policy.

Testing

  • IcebergJdbcMetaStorePropertiesTest#uriAliasesAreMaskedWithoutChangingConnectionValues: fails its credential-rendering assertion before the URI fix and passes afterward.
  • Local targeted FE/connector tests: 277 passed, zero failures, errors, or skips; all 50 required XML reports were fresh. Includes the full connector SPI suite and the affected properties, provider surfaces, metastore dispatch, registration, display, and audit tests.
  • Local isolated 1 FE + 3 BE, ASAN: query_p0/show/test_catalog_credential_masking.groovy passed CREATE/ALTER checks across four metastore configurations: 24 display checks and nine unauthorized checks. Callback-external SQL counters confirmed execution; the runner's verdict remains PASS_WITH_COVERAGE_CAVEATS.
  • Ordinary JDBC SQL probe: 12 display checks passed for jdbc_url and jdbc.jdbc_url after CREATE and ALTER, including an address without credentials, using local MySQL Connector/J 8.0.33.
  • Fake credentials and test_connection=false avoid external services. These are targeted local runs, without external metastore I/O, real Kerberos authentication, full P0, or community CI.

Notes for Reviewers

The plugin SPI and API 32.0 remain unchanged. The committed regression requires no external driver fixture; ordinary JDBC SQL verification used a locally supplied driver.

Release note

Catalog displays mask full JDBC/REST URI values and their aliases, including addresses without embedded credentials. Connections continue to receive the original URI values.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Ready

AI Disclosure

  • Tool: Codex (GPT-6)
  • Generated: implementation, tests, verification scripts, and this description
  • Human review: not stated
  • Prompts: Asked to reuse upstream masking registration, keep keytab paths visible, and fix URI omissions with unit tests and SELECT-only checks through all three display paths in an isolated FE/BE cluster.

### Background

Catalog display commands render stored connection properties through a shared
sensitive-key set.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#61094 (introduced Paimon JDBC support)
PR Type: Bugfix

Problem Summary:

Catalog readers can see unmasked metastore credentials and credential-bearing
connection strings when stored properties use aliases missing from the shared
masking set. Register the omitted password, DLF and JDBC/REST URI keys so all
catalog display paths mask them without changing connection properties.

### Details

- **Frequency:** Always when omitted properties are stored and displayed.
- **Root cause:** Sensitive metastore aliases are absent from display masking.
- **Why existing tests missed it:** They cover generic passwords and existing
  storage/OAuth keys, without metastore aliases or SELECT-only display access.
- **Fix:** Extend the existing case-insensitive display set. Mask complete URIs,
  including endpoints without embedded credentials.
- **Other paths and branches:** SHOW CREATE CATALOG, SHOW CATALOG and catalogs()
  share this behavior. Release branches require separate adaptation to their
  legacy PrintableMap implementation and have not been tested here.
- **Behavior change:** The listed connection URI properties display as *XXX.

### Testing

Three regression tests fail on the baseline and pass with the fix. All 26
selected FE tests pass with zero failures, errors or skips, including four
catalog property fixtures under real SELECT-only and denied privilege checks.
Checkstyle and git diff --check pass. Verification uses embedded FE command
and metadata paths, without a live external metastore or full FE/BE cluster.

### Release note

Mask metastore JDBC passwords, Paimon DLF credentials and JDBC/REST connection
URIs in catalog displays.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex
- Generated: Code, tests and this message
- Human review: Pending
新增四类真实 Catalog 的 SQL 回归,用假凭证检查 SHOW CREATE CATALOG、
SHOW CATALOG 和 catalogs() 的脱敏,并核对仅 SELECT 授权及无授权行为。

验证:隔离 1 FE + 3 BE,P0 定向 test_catalog_credential_masking 通过。
### Background

Catalog 保存原始 metastore 属性,三个展示入口共用敏感键集合。插件属性上的 sensitive 声明此前只供日志使用,未接入展示层。

### What problem does this PR solve?

Issue Number: None
Related PR: apache#61094 (Paimon JDBC credential aliases)
PR Type: Bugfix

Problem Summary:

仅有 Catalog SELECT 权限的用户可能读取未列入静态集合的敏感属性。通过公共插件契约在启动时注册 metastore 的全部敏感别名,并补齐 Iceberg JDBC 的密码导出。两个 HMS keytab 路径随声明自动参与脱敏。

### Details

- **Change map:** 生产代码增加 71 行、删除 7 行;测试增加 381 行、删除 39 行;接口记录增加 1 行;开发文档改动 7 行,API 版本配置改动 2 行。主要改动用于验证自动发现、插件注册、权限和重放。
- **Frequency:** Always;已保存的敏感键仅由 metastore provider 导出、未进入展示集合时发生。
- **Root cause:** ConnectorPluginManager 没有接收 metastore 的敏感键;Iceberg JDBC provider 继承默认空导出。
- **Why existing tests missed it:** 展示类测试和此前的四类 SQL 回归只覆盖已列出的静态键,没有新敏感别名或 HMS keytab 路径。注册器测试没有覆盖敏感键传递。
- **Fix:** MetaStoreProviders 汇总当前插件类加载器发现的后端别名;Hive、Hudi、Iceberg、Paimon provider 通过公共 SPI 导出。FE 在发布 provider 前注册敏感键,展示时复用已有集合。导出失败的目录插件被拒绝,且不占用类型名。
- **Other paths and branches:** 同时覆盖 SHOW CREATE CATALOG、SHOW CATALOG 和 catalogs()。本次目标为 master;发布分支的旧展示及插件架构未执行回补验证。
- **Behavior change:** provider 声明的敏感属性及别名在三个展示入口统一掩码,包括两个 HMS keytab 路径。权限判断、属性原值和普通选项保持原样。
- **Compatibility:** Connector 插件 API 从 31.0 升为 32.0,插件需按新版本重新构建。元数据存储格式不变;注册只读取属性声明,不绑定 Catalog 或访问外部服务。

### Testing

- 未修复代码上的两个新增测试分别因敏感键未掩码、Iceberg JDBC 导出为空而失败;均为目标断言失败。
- 最终 226 项定向测试通过,0 失败、0 错误、0 跳过。包含完整 SPI 套件,以及自动别名、类加载器、注册故障隔离和 FE 展示测试。
- 原生 P0 回归在隔离 1 FE + 3 BE 中通过:七组 Catalog 配置、30 个正向入口检查、15 个无授权检查;覆盖新建和 ALTER。
- JDBC 独立 SQL 探针的三个展示入口通过。三个 HMS Catalog 和原 SELECT 账号在集群重启后继续通过九个入口检查,未重建元数据。
- 实际 BE numbers 扫描求和为 499500。联合构建、Checkstyle、接口基线及补丁检查通过。
- 定向验证未替代全量 P0、社区 CI 或外部 metastore 读写测试。

### Notes for Reviewers

敏感键通过插件接口传递,fe-core 不依赖具体 metastore 实现。两份接口基线均从实际编译类重新生成,只有 provider 新方法改变;metadata 基线内容保持一致。

### Release note

Catalog 展示自动掩码 metastore provider 声明的敏感属性和别名,包括 HMS keytab 路径。Connector 插件 API 升至 32.0。

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No;开发指南已同步插件敏感键导出规则。
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex
- Generated: 修复、测试和说明。
- Human review: 待作者审阅。
### Background

Catalogs store JDBC and REST connection URIs, which can include credentials without a separate password property. Connector providers export sensitive property names and aliases at FE startup; catalog displays use the shared registry to mask their values.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70338 (added connector-declared masking and metastore registration)
PR Type: Bugfix

Problem Summary:

Catalog readers with SELECT can still see credentials embedded in JDBC and REST URI properties because those properties are not declared sensitive. Mark the five URI fields sensitive so the existing registration chain covers their aliases, including the legacy JDBC prefix. Most of the change adds checks for aliases, preserved connection values, and SELECT-only display access.

### Details

- **Frequency:** Always when an affected URI contains credentials and a catalog reader displays its properties.
- **Root cause:** The JDBC and Iceberg/Paimon JDBC/REST URI annotations omit `sensitive=true`, so their names never reach the provider's masking surface.
- **Why existing tests missed it:** Property tests covered binding and standalone passwords; provider tests omitted URI keys, and the Iceberg dispatch test explicitly expected `uri` to remain unmasked. The existing audit regression checks standalone passwords and audit text, without SELECT-only URI alias checks.
- **Fix:** Declare URI sensitivity in the property holders and reuse the existing alias expansion, metastore union, and provider registration. Mask complete URI values because JDBC and REST support different credential encodings.
- **Other paths and branches:** Covers `SHOW CREATE CATALOG`, `SHOW CATALOG <name>`, and `catalogs()`, plus annotation-based property masking. Both `jdbc_url` and `jdbc.jdbc_url` are covered. Only master was verified; release-branch applicability and the earliest introducing changes remain unconfirmed.
- **Behavior change:** These URI values are masked even without embedded credentials. The shared `uri` alias is masked wherever the common registry is used. Stored connection values and ordinary properties retain their behavior; keytab paths follow the upstream visibility policy.

### Testing

- `IcebergJdbcMetaStorePropertiesTest#uriAliasesAreMaskedWithoutChangingConnectionValues`: fails its credential-rendering assertion before the URI fix and passes afterward.
- Local targeted FE/connector tests: 277 passed, zero failures, errors, or skips; all 50 required XML reports were fresh. Includes the full connector SPI suite and the affected properties, provider surfaces, metastore dispatch, registration, display, and audit tests.
- Local isolated 1 FE + 3 BE, ASAN: `query_p0/show/test_catalog_credential_masking.groovy` passed CREATE/ALTER checks across four metastore configurations: 24 display checks and nine unauthorized checks. Callback-external SQL counters confirmed execution; the runner's verdict remains `PASS_WITH_COVERAGE_CAVEATS`.
- Ordinary JDBC SQL probe: 12 display checks passed for `jdbc_url` and `jdbc.jdbc_url` after CREATE and ALTER, including an address without credentials, using local MySQL Connector/J 8.0.33.
- Fake credentials and `test_connection=false` avoid external services. These are targeted local runs, without external metastore I/O, real Kerberos authentication, full P0, or community CI.

### Notes for Reviewers

The plugin SPI and API 32.0 remain unchanged. The committed regression requires no external driver fixture; ordinary JDBC SQL verification used a locally supplied driver.

### Release note

Catalog displays mask full JDBC/REST URI values and their aliases, including addresses without embedded credentials. Connections continue to receive the original URI values.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: implementation, tests, verification scripts, and this description
- Human review: not stated
- Prompts: Asked to reuse upstream masking registration, keep keytab paths visible, and fix URI omissions with unit tests and SELECT-only checks through all three display paths in an isolated FE/BE cluster.
### Background

The catalog credential masking suite checks three display paths with low-privilege
users. The catalogs table-valued function executes a metadata scan on a backend.

### What problem does this PR solve?

Issue Number: None

Cloud P0 needs a compute group for each test user before it can execute the
metadata scans. The catalog SELECT assertion also needs to isolate catalog
privileges from execution resource grants.

### Details

- **Failure:** The catalog masking suite's Cloud metadata scan requires a compute group.
- **Cause:** The reader and ungranted user had no compute group usage. The earlier grant check combined object and execution resource privileges.
- **Stability evidence:** The exact same suite passed on isolated Cloud and shared-nothing clusters. Each run executed 24 reader checks and 9 access-denied checks. The Cloud run verified USAGE for both users.

### Testing

- Ran `test_catalog_credential_masking` under the native Cloud P0 and shared-nothing P0 profiles with one selected suite passing in each run.
- Verified SELECT-only grants for all four target catalogs in both runs and compute group USAGE for both test users in Cloud.

### Release note

None.

### Check List (For Author)

- Behavior changed: No.
- Does this need documentation: No.
- Pre-submission status: Ready.

### AI Disclosure

- Tool: OpenAI Codex.
- Generated: Regression suite update and commit description.
- Human review: Pending.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70696
commit: a3bbe0d822e602557f0ad79d7bf7ac04b9c34007
base: e04a785e20a87ef910ec756ff6b4d269298addb6
reviewed_at: 2026-11-18T03:33+08:00
reviewer: OIiveirra
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • regression-test/suites/query_p0/show/test_catalog_credential_masking.groovy:54-83 — Prior F-01 is fixed: both users receive Cloud compute-group USAGE, and target catalog SELECT privileges are asserted separately from execution resources.
  • regression-test/suites/query_p0/show/test_catalog_credential_masking.groovy:99-167 — All 24 reader and nine denied-access checks remain. Existing author-side isolated shared-nothing and Cloud receipts match the committed suite; these are targeted runs, not full P0 or community CI.
  • Full 12-file re-review found no new issues. The review team performed source review only and did not run builds, tests, or cluster operations.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant