Skip to content

[FLINK-39844][postgres] Filter snapshot table columns by schema name to fix LIKE-wildcard cross-schema column leakage - #4435

Open
JNSimba wants to merge 2 commits into
apache:masterfrom
JNSimba:fix-pg-snapshot-schema-like-match
Open

JNSimba wants to merge 2 commits into
apache:masterfrom
JNSimba:fix-pg-snapshot-schema-like-match

Conversation

@JNSimba

@JNSimba JNSimba commented Jun 4, 2026 •

Copy link
Copy Markdown
Member

https://issues.apache.org/jira/browse/FLINK-39844

What is the purpose of the change

During the snapshot phase, the PostgreSQL connector reads column structure via JDBC DatabaseMetaData#getColumns(catalog, schemaPattern, tableNamePattern, columnNamePattern). Per the JDBC spec, both schemaPattern and tableNamePattern are LIKE patterns, where _ matches any single character and % matches any sequence of characters. Both are legal identifier characters in PostgreSQL, so getColumns can return columns from other schemas/tables that were never meant to match.

An exact filter on the table name already exists, but the schema name was never validated. When two schemas have names that are wildcard matches of each other (e.g. sch_test and schxtest, where _ matches x) and both contain a same-named table, capturing sch_test.<table> also pulls in the look-alike schema's columns. The table-name filter cannot tell them apart, so the snapshot fails with IllegalStateException: Duplicate key Optional.empty (or, when columns differ, silently merges columns from the wrong schema).

Brief change log

  • PostgresConnection#doReadTableColumn: also compare the result-set TABLE_SCHEM (column 2) against TableId.schema(), in addition to the existing TABLE_NAME check. The schema check is skipped when the requested schema is null, so columns are not dropped when no schema is specified. This is a pure after-the-fact filter; it does not change the metadata query and does not affect normal (non-wildcard) schemas/tables.
  • SimilarTableNamesITCase / similar_names.sql: add a cross-schema case with two schemas (sch_test / schxtest) that are wildcard matches of each other, each holding a same-named table, verifying that only the target schema's snapshot and incremental data are captured.

Verifying this change

This change added tests and can be verified as follows:

  • Added SimilarTableNamesITCase#testReadTableWithSimilarSchemaNameUnderscore, which fails (snapshot IllegalStateException: Duplicate key Optional.empty) without the fix and passes with it.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: no
  • The connector code base: yes (postgres-cdc snapshot column reading)

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

…to fix LIKE-wildcard cross-schema column leakage
@JNSimba
JNSimba force-pushed the fix-pg-snapshot-schema-like-match branch from 49613db to 75f4181 Compare June 4, 2026 07:48

@Mrart Mrart left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need add ut in ‘PostgresPipelineITCase’ to cover this change

@JNSimba

JNSimba commented Aug 20, 2026

Copy link
Copy Markdown
Member Author

I think we need add ut in ‘PostgresPipelineITCase’ to cover this change

Should SimilarTableNamesITCase.java be able to override this? This is a source-level change, so perhaps we don't need to add a case to the pipeline?

// - When querying 'user_sink', the pattern may also match 'userbsink' (due to '_')
// - When querying 'user%data' (where % is literal), it may match 'user_test_data' (due to
// '%')
// Both the schema name and table name are passed to DatabaseMetaData#getColumns as LIKE

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix — I traced the mechanism end to end and the change is correct. One request on how we record it, plus some context that may save you a round trip.

Upstream Debezium already fixed this, but differently. It's DBZ-9305 "Fix pgoutput decoder metadata query with wildcard characters", first released in 3.3.0.Final. Instead of post-filtering the result set, it escapes the LIKE pattern at the call site

final String schemaNamePattern = createPatternFromName(tableId.schema(), databaseMetaData.getSearchStringEscape());
final String tableNamePattern  = createPatternFromName(tableId.table(),  databaseMetaData.getSearchStringEscape());
try (ResultSet columnMetadata = databaseMetaData.getColumns(null, schemaNamePattern, tableNamePattern, null)) {

Their test (PostgresConnectorIT#shouldStreamSuccessfullyWithSchemasWithUnderscores) uses schemas test_ / test1 each holding a table tab — essentially the same shape as your sch_test / schxtest, so it's good confirmation we converged on the same bug.

We can't take the upstream approach on our current pin, so your approach is the right one here. Two reasons: createPatternFromName lives in debezium-core's JdbcConnection and doesn't exist in the 1.9.8.Final we pin; and the buggy call site (PgOutputMessageDecoder#getTableColumnsFromDatabase) is not one of the Debezium classes we copy into this repo, so escaping there would mean pulling in a ~700-line class. Post-filtering inside the already-copied PostgresConnection#doReadTableColumn is the minimal option.

What I'd like changed: please record this in the class-level javadoc's "Copied from Debezium 1.9.8-Final with the following modifications" list, the same way the existing DBZ-5398 entry does ("Remove this comment after bumping debezium version to 2.0.0.Final"). Right now the FLINK-38965 entry is there but this change adds nothing, so both filters read as permanent local behaviour rather than tracked debt that DBZ-9305 subsumes. Suggested entry

    FLINK-39844: Modified doReadTableColumn to also filter out columns from other schemas
        returned due to PostgreSQL LIKE wildcard matching on the schema name, e.g. querying
        schema 'sch_test' also matches 'schxtest'. Upstream fixed the same issue in DBZ-9305
        (Debezium 3.3.0.Final) by escaping the LIKE pattern via
        JdbcConnection#createPatternFromName before calling DatabaseMetaData#getColumns, which
        we cannot backport because the call site lives in PgOutputMessageDecoder. Remove this
        and the FLINK-38965 filter after bumping debezium version to 3.3.0.Final.

One more thing worth a line in the code comment, because it confused me at first: of the two callers of doReadTableColumn, only readColumnForDecoder is actually protected by this filter. The other one (readTableColumn, reached from JdbcConnection#getColumnsDetails) is passed a TableId built from the same* result-set row (getString(2) / getString(3)), so both the existing table-name check and your new schema check are tautologies on that path — it's covered by tableFilter.isIncluded(tableId) instead. Noting this stops the next reader from thinking the filter is redundant, or "simplifying" it away.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JNSimba would you like to push fowards this pr, and it will be merged this week.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@loserwang1024 loserwang1024 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants