Repository navigation
[FLINK-39844][postgres] Filter snapshot table columns by schema name to fix LIKE-wildcard cross-schema column leakage - #4435
Conversation
…to fix LIKE-wildcard cross-schema column leakage
49613db to
75f4181
Compare
Mrart
left a comment
There was a problem hiding this comment.
I think we need add ut in ‘PostgresPipelineITCase’ to cover this change
Should SimilarTableNamesITCase.java be able to override this? This is a source-level change, so perhaps we don't need to add a case to the pipeline? |
| // - When querying 'user_sink', the pattern may also match 'userbsink' (due to '_') | ||
| // - When querying 'user%data' (where % is literal), it may match 'user_test_data' (due to | ||
| // '%') | ||
| // Both the schema name and table name are passed to DatabaseMetaData#getColumns as LIKE |
There was a problem hiding this comment.
Thanks for the fix — I traced the mechanism end to end and the change is correct. One request on how we record it, plus some context that may save you a round trip.
Upstream Debezium already fixed this, but differently. It's DBZ-9305 "Fix pgoutput decoder metadata query with wildcard characters", first released in 3.3.0.Final. Instead of post-filtering the result set, it escapes the LIKE pattern at the call site
final String schemaNamePattern = createPatternFromName(tableId.schema(), databaseMetaData.getSearchStringEscape());
final String tableNamePattern = createPatternFromName(tableId.table(), databaseMetaData.getSearchStringEscape());
try (ResultSet columnMetadata = databaseMetaData.getColumns(null, schemaNamePattern, tableNamePattern, null)) {Their test (PostgresConnectorIT#shouldStreamSuccessfullyWithSchemasWithUnderscores) uses schemas test_ / test1 each holding a table tab — essentially the same shape as your sch_test / schxtest, so it's good confirmation we converged on the same bug.
We can't take the upstream approach on our current pin, so your approach is the right one here. Two reasons: createPatternFromName lives in debezium-core's JdbcConnection and doesn't exist in the 1.9.8.Final we pin; and the buggy call site (PgOutputMessageDecoder#getTableColumnsFromDatabase) is not one of the Debezium classes we copy into this repo, so escaping there would mean pulling in a ~700-line class. Post-filtering inside the already-copied PostgresConnection#doReadTableColumn is the minimal option.
What I'd like changed: please record this in the class-level javadoc's "Copied from Debezium 1.9.8-Final with the following modifications" list, the same way the existing DBZ-5398 entry does ("Remove this comment after bumping debezium version to 2.0.0.Final"). Right now the FLINK-38965 entry is there but this change adds nothing, so both filters read as permanent local behaviour rather than tracked debt that DBZ-9305 subsumes. Suggested entry
FLINK-39844: Modified doReadTableColumn to also filter out columns from other schemas
returned due to PostgreSQL LIKE wildcard matching on the schema name, e.g. querying
schema 'sch_test' also matches 'schxtest'. Upstream fixed the same issue in DBZ-9305
(Debezium 3.3.0.Final) by escaping the LIKE pattern via
JdbcConnection#createPatternFromName before calling DatabaseMetaData#getColumns, which
we cannot backport because the call site lives in PgOutputMessageDecoder. Remove this
and the FLINK-38965 filter after bumping debezium version to 3.3.0.Final.
One more thing worth a line in the code comment, because it confused me at first: of the two callers of doReadTableColumn, only readColumnForDecoder is actually protected by this filter. The other one (readTableColumn, reached from JdbcConnection#getColumnsDetails) is passed a TableId built from the same* result-set row (getString(2) / getString(3)), so both the existing table-name check and your new schema check are tautologies on that path — it's covered by tableFilter.isIncluded(tableId) instead. Noting this stops the next reader from thinking the filter is redundant, or "simplifying" it away.
There was a problem hiding this comment.
@JNSimba would you like to push fowards this pr, and it will be merged this week.
https://issues.apache.org/jira/browse/FLINK-39844
What is the purpose of the change
During the snapshot phase, the PostgreSQL connector reads column structure via JDBC
DatabaseMetaData#getColumns(catalog, schemaPattern, tableNamePattern, columnNamePattern). Per the JDBC spec, bothschemaPatternandtableNamePatternare LIKE patterns, where_matches any single character and%matches any sequence of characters. Both are legal identifier characters in PostgreSQL, sogetColumnscan return columns from other schemas/tables that were never meant to match.An exact filter on the table name already exists, but the schema name was never validated. When two schemas have names that are wildcard matches of each other (e.g.
sch_testandschxtest, where_matchesx) and both contain a same-named table, capturingsch_test.<table>also pulls in the look-alike schema's columns. The table-name filter cannot tell them apart, so the snapshot fails withIllegalStateException: Duplicate key Optional.empty(or, when columns differ, silently merges columns from the wrong schema).Brief change log
PostgresConnection#doReadTableColumn: also compare the result-setTABLE_SCHEM(column 2) againstTableId.schema(), in addition to the existingTABLE_NAMEcheck. The schema check is skipped when the requested schema isnull, so columns are not dropped when no schema is specified. This is a pure after-the-fact filter; it does not change the metadata query and does not affect normal (non-wildcard) schemas/tables.SimilarTableNamesITCase/similar_names.sql: add a cross-schema case with two schemas (sch_test/schxtest) that are wildcard matches of each other, each holding a same-named table, verifying that only the target schema's snapshot and incremental data are captured.Verifying this change
This change added tests and can be verified as follows:
SimilarTableNamesITCase#testReadTableWithSimilarSchemaNameUnderscore, which fails (snapshotIllegalStateException: Duplicate key Optional.empty) without the fix and passes with it.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation