Skip to content

[FLINK-41197][connect/mysql] Filter schema changes by captured table IDs - #4601

Draft
txjytxwd wants to merge 1 commit into
apache:masterfrom
txjytxwd:fix/flink-41197-filter-multi-table-drop-master
Draft

txjytxwd wants to merge 1 commit into
apache:masterfrom
txjytxwd:fix/flink-41197-filter-multi-table-drop-master

Conversation

@txjytxwd

@txjytxwd txjytxwd commented Oct 26, 2026 •

Copy link
Copy Markdown

What is the purpose of this pull request?

Fix FLINK-41197. With scan.binlog.newly-added-table.enabled=true, MySQL can report the names affected by a multi-table DROP TABLE as an aggregated source table name. Matching that name can bypass tables.exclude, allowing a DropTableEvent for an excluded table to reach downstream operators and cause a Transform failure.

For DROP TABLE keep_multi, excluded_multi, the pipeline should emit only the event for keep_multi when excluded_multi is excluded. A statement affecting only excluded tables should not be emitted.

Brief change log

  • Resolve the actual table IDs for aggregated schema records using the existing MySQL DDL parser before applying the binlog reader table filter. Preserve the existing fallback when table identities cannot be resolved.
  • Apply the configured table filter to each deserialized Pipeline schema change event. Keep the original deserializer responsible for parsing and preserve data event, type information, and collector behavior.
  • Add regression coverage for mixed and fully excluded multi-table DDL, qualified and quoted identifiers, and existing schema-processing behavior.

Verifying this change

  • BinlogSplitReaderFilteringTest covers admission of schema change records.
  • MySqlPipelineRecordEmitterTest covers filtering the actual emitted events using the real MySQL deserializer.
  • Related existing MySQL record-emitter, record-utility, and source-config tests were also exercised under the default and flink2 profiles.
  • The final diff audit verified all changed behaviors with targeted executed tests and explicit exclusions. Spotless, Checkstyle, and ASF license-header checks passed for the new tests. The reporter also reproduced the original failure using the CDC 3.6.0 MySQL connector with matching common/runtime/flink1-compat JARs on Flink 1.20.
  • The release-3.6 implementation of this fix was manually validated on Flink 1.20 using package 3.6.0-flink-41197 from commit 6141b76d. The reporter reran the original DDL with a Print sink and confirmed that the job continued running normally, with no excluded-table event or original Transform NPE in the supplied logs. Direct end-to-end validation of this master commit remains pending.

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable
Was generative AI tooling used to co-author this PR?
  • Yes (Codex)

Generated-by: Codex

Fix FLINK-41197 by applying table filters to multi-table DDL and
individual Pipeline schema change events.

Co-Authored-By: Flux <flux@bytedance.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant