Skip to content

[FLINK-41485][common] Support column reorder events in CDC pipelines - #4626

Open
lvyanquan wants to merge 3 commits into
apache:masterfrom
lvyanquan:codex/flink-41485-column-reordering
Open

lvyanquan wants to merge 3 commits into
apache:masterfrom
lvyanquan:codex/flink-41485-column-reordering

Conversation

@lvyanquan

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

Fix FLINK-41485: propagate MySQL column reordering so downstream schemas stay aligned with the layout of subsequent records. Ignoring MODIFY COLUMN / CHANGE COLUMN with FIRST or AFTER can otherwise silently assign values to the wrong fields.

JIRA: https://issues.apache.org/jira/browse/FLINK-41485

Brief change log

  • Add validated column-move events, schema application and serialization. Preserve existing event identifiers and the legacy add-column API/wire format.
  • Emit reordering events in DDL order, including compound changes, rename/type changes and case normalization.
  • Support synchronous and asynchronous transforms, computed columns, routing, and record remapping when reordering is excluded, lenient or unsupported in tolerant mode.
  • Update Kafka/Values schemas and physically reorder Paimon/StarRocks columns. Preserve column definitions and enforce StarRocks key-order constraints; explicitly reject unsupported moves in Doris, Fluss, Iceberg, MaxCompute and OceanBase.
  • Document configuration and sink behavior in English and Chinese.

Verifying this change

  • 194 targeted tests passed with Flink 1.20.3 / Java 11, and the same 194 tests passed with Flink 2.2.0 / Java 17.
  • Kafka JSON regression tests verify that same-type name / city fields retain their values after reordering.
  • Regression proof: restoring the master MySQL parser makes 8 of the 10 parser tests fail; restoring the fix makes all 10 pass.
  • Added MySQL-to-Values / MySQL-to-Kafka end-to-end coverage and StarRocks integration coverage. E2E sources compile under Flink 1.20.3 / Java 11; Docker-backed integration and E2E cases have not been executed locally.
  • Spotless and Checkstyle passed for core, CLI and affected connector modules; the E2E module also passed Spotless. GitHub CI has not run yet.

Documentation

  • Does this pull request introduce a new feature? Yes, column-reordering events and their pipeline handling.
  • How is the feature documented? Docs and JavaDocs, including English and Chinese schema-evolution documentation.

Was generative AI tooling used to co-author this PR?
  • Yes — Codex CLI 0.163.0-alpha.10.2

Generated-by: Codex CLI 0.163.0-alpha.10.2

ruanhang.ruan and others added 2 commits November 17, 2026 14:48
Port column reorder events and adapt schema visitors, serializers, synchronous
and asynchronous transforms, and connector handling to the upstream codebase.
Preserve existing event identifiers and legacy add-column serialization.

Generated-by: Codex CLI 0.163.0-alpha.10.2

AI-Contributed/Feature: 1583/1583
AI-Contributed/UT: 3758/3758
Use same-type name/city columns to verify that reordering preserves their
association with values, instead of merely catching incompatible layouts.

Generated-by: Codex CLI 0.163.0-alpha.10.2

AI-Contributed/Feature: 0/0
AI-Contributed/UT: 24/24
@lvyanquan

Copy link
Copy Markdown
Contributor Author

@sd4324530 CC as you may be interested in this.

Use fragment matching while waiting for initial table schemas, and align
Kafka Debezium JSON expectations with upstream source metadata.

Generated-by: Codex CLI 0.163.0-alpha.10.2
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 16/16

@sd4324530 sd4324530 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on FLINK-41485. The overall approach is solid: a validated, immutable ReorderColumnEvent, transform-side move derivation, and per-sink handling with documentation and tests is the right direction, and it fixes the basic MySQL → Paimon/Kafka misalignment case.

I left 14 inline comments. Before merging, I would like the following blockers addressed:

  1. Non-asterisk transforms still silently misalign. The reorder event is dropped in the non-asterisk pre-transform path while PreTransformOperator rebuilds its processor from the reordered source schema, so downstream rows change order without a schema update. Please either forward a derived pre-transform reorder event or keep the output order stable, and add a non-asterisk projection test.
  2. StarRocks MODIFY ... AFTER fails deterministically. The AlterColumnTypeEvent starts an asynchronous StarRocks schema-change job, and the immediately following reorder ALTER is rejected while the table is in SCHEMA_CHANGE. Please wait for the type job (like executeAlter) or combine type and position into one ALTER, and add a type + reorder integration test.
  3. Hudi cannot tolerate reorder in TRY_EVOLVE. The 8-arg visitor default throws UnsupportedSchemaChangeEventException, but HudiMetadataApplier wraps it in a plain RuntimeException, so the coordinator does not ignore it. Please use the 10-arg overload, do not wrap SchemaEvolveException, and add Hudi to the documentation.
  4. Excluding column changes can break reorder. With exclude.schema.changes: [add.column] (or rename.column) but reorder.column included, the evolved schema lacks the referenced column and the job fails even in TRY_EVOLVE. Please filter/rewrite moves against the evolved schema, or throw UnsupportedSchemaChangeEventException, and cover this configuration in a test.

The remaining inline comments are smaller suggestions: StarRocks job amplification and connection/retry handling, unsupported-move exception types, the information_schema column-order assumption, MySQL identifier casing, LCS memory, duplicated move logic, test coverage for metadata preservation, and documentation/upgrade notes.

Disclosure: this review was primarily generated by an AI assistant (Claude) from a static review of the PR diff, and was reviewed and confirmed by me before posting. I did not run the new unit/integration tests locally, so the transform and StarRocks scenarios in particular should be validated with targeted tests.

return position;
}

public org.apache.flink.cdc.common.event.ColumnPosition getColumnPosition() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: getColumnPosition() dereferences position without a null check, but the still-public deprecated constructor at line 130 accepts a null position. Code that previously relied on the null-safe AddColumnEvent.ColumnPosition.LAST.equals(e.getPosition()) check in SchemaUtils.transformSchemaChangeEvent will now throw an NPE.

Suggestion: Make this method null-safe (return null, or checkNotNull with a clear message), or validate position in the deprecated constructor. Low likelihood, but it is a behavior change.


if (hasAsteriskMap.getOrDefault(tableId, true)) {
// See comments in PreTransformOperator#cacheChangeSchema method.
if (event instanceof ReorderColumnEvent) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: Non-asterisk projections still silently misalign. When hasAsteriskMap is false, PreTransformOperator.cacheChangeSchema calls SchemaUtils.transformSchemaChangeEvent(false, ...), which returns Optional.empty() for ReorderColumnEvent (SchemaUtils.java:379-383), so the event never reaches PostTransformOperator. However, cachePreTransformProcessor (PreTransformOperator.java:259) rebuilds the pre-transform processor from the reordered source schema, so the rows emitted by PreTransformOperator change order while this operator pre-schema stays the same. Every subsequent record is then decoded positionally with the old schema, and values are silently swapped.

Suggestion: Handle reorder in the non-asterisk branch as well — for example, after rebuilding the processor, derive oldPreSchema/newPreSchema and forward ReorderColumnEventTransformer.transformReorderColumnEvent(...), or keep the pre-transform output order stable. Add a test with a non-asterisk projection such as id, age and a move of age; the current test only covers * (testReorderWithPreWildcardAndComputedColumn).

}
}

private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: MySQL parses MODIFY COLUMN ... AFTER into an AlterColumnTypeEvent followed by this ReorderColumnEvent (CustomAlterTableParserListener.java:450-471). applyAlterColumnType (line 324) calls StarRocksEnrichedCatalog.alterColumnType, which submits an asynchronous StarRocks schema-change job and returns without waiting (StarRocksEnrichedCatalog.java:121-152). This method then immediately issues the position ALTER. StarRocks allows only one schema change per table, so the second ALTER is rejected with SCHEMA_CHANGE, and the job fails deterministically. This also happens for a pure position change, because the type event is always emitted.

Suggestion: Make alterColumnType wait for its job like executeAlter, wait for any in-flight column alter job before the first move, or combine type and position into one ALTER. Add a test that applies AlterColumnTypeEvent and then ReorderColumnEvent; the current IT only sends the reorder.

renameColumnEventVisitor,
truncateTableEventVisitor,
alterTableCommentEventVisitor,
reorderColumnEvent -> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: The 8-argument overloads now throw UnsupportedSchemaChangeEventException for reorder, but the only in-tree caller, HudiMetadataApplier (HudiMetadataApplier.java:123-160), catches Exception and wraps it in a plain RuntimeException. SchemaCoordinator.shouldIgnoreException only tolerates UnsupportedSchemaChangeEventException in TRY_EVOLVE, so Hudi jobs fail instead of keeping the sink schema. Hudi is also missing from the new documentation list.

Suggestion:

  • Update Hudi to use the 10-arg overload with an explicit reorder -> throw new UnsupportedSchemaChangeEventException(event) handler, and do not wrap SchemaEvolveException.
  • Add Hudi to the docs.
  • Consider deprecating the 8-arg overloads so third-party sinks migrate.

return candidate;
}
}
throw new IllegalArgumentException(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: This throws for a moved column (line 253 does the same for a missing anchor) that is absent from the evolved schema. That happens for a supported configuration: exclude add.column (or rename.column) but keep reorder.column. The original schema has the column, the evolved schema does not, and applying the reorder fails here. SchemaCoordinator.shouldIgnoreException does not tolerate IllegalArgumentException, so even TRY_EVOLVE fails. The new E2E testExcludedReorderAndAddColumns excludes both types, so it does not cover this.

Suggestion: Rewrite/filter reorder moves against the current evolved schema in SchemaDerivator.normalizeSchemaChangeEvents (drop moves whose moved or reference column is absent, and drop the event if nothing remains), or throw UnsupportedSchemaChangeEventException. Add a test that excludes only add.column.


private static List<String> longestCommonSubsequence(
List<String> oldColumnNames, List<String> newColumnNames) {
int[][] lengths = new int[oldColumnNames.size() + 1][newColumnNames.size() + 1];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: This allocates an (n+1) x (m+1) int matrix per transform instance per reorder event. For a 10,000-column table that is about 400 MB and about 100M comparisons; 1,000 columns is about 4 MB per instance.

Suggestion: Since the two schemas are permutations of the same column set, compute the LIS over the old-index to new-index permutation in O(n log n), or derive moves directly from the event. At minimum, guard the matrix size.

newColumnPositions.put(newColumnNames.get(i), i);
}
List<ReorderColumnEvent.ColumnMove> columnMoves = new ArrayList<>();
if (stableAnchors.isEmpty()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: This branch is unreachable: two non-empty permutations of the same column set always have an LCS of at least one element, and equal lists return at line 45. It has no test coverage.

Suggestion: Remove it. The targetIndex = 0 initialization in the following loop already handles the first-anchor case.

private boolean applyColumnMove(
List<String> simulatedColumnNames, ReorderColumnEvent.ColumnMove columnMove) {
List<String> reorderedColumnNames = new ArrayList<>(simulatedColumnNames);
checkArgument(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: Move application is now implemented four times with slightly different semantics: SchemaUtils.applyReorderColumnEvent (SchemaUtils.java:192-259), this method, StarRocksMetadataApplier.applyColumnMoves (lines 438-491), and ReorderColumnEventTransformer. Missing columns here throw IllegalArgumentException, which TRY_EVOLVE does not tolerate.

Suggestion: Extract a shared helper (for example applyMoves(List<String>, List<ColumnMove>) returning the reordered names, or a normalized/minimal move list on ReorderColumnEvent) so no-op handling and missing-column policy stay consistent. Use UnsupportedSchemaChangeEventException for unsupported moves.

}

@Test
void testReorderColumnSupportsMovesAndPreservesMetadata() throws Exception {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: This test cannot catch metadata loss: MockStarRocksCatalog.alterColumnPosition (MockStarRocksCatalog.java:200) ignores the passed StarRocksColumn/isKey and just reorders the existing column objects, so dropping DEFAULT/nullability/COMMENT in loadColumnDefault/buildAlterColumnPositionSql leaves it green. The IT (StarRocksMetadataApplierITCase.java:737) also sends only the reorder event and has no defaults/comments.

Suggestion: Make the mock apply the restated definition (or assert on real information_schema values), and add an IT with DEFAULT/COMMENT and a preceding AlterColumnTypeEvent. Also cover defaults like CURRENT_TIMESTAMP(3), which formatDefaultValue currently quotes as a string.


The MySQL pipeline source emits `ReorderColumnEvent` for `MODIFY COLUMN` and `CHANGE COLUMN` statements with `FIRST` or `AFTER` clauses. Renaming, type changes, and column moves in a compound DDL are applied in statement order. The `reorder` and `column` families both include `reorder.column`.

Kafka updates its serialization schema. Paimon applies physical column moves, while StarRocks only accepts moves that keep primary key columns first and in their existing order. Doris, Fluss, Iceberg, MaxCompute, and OceanBase reject physical column reordering explicitly. In `EVOLVE` mode, an unsupported move fails the job rather than silently misaligning field values. In `TRY_EVOLVE` mode, the framework retains the sink schema and maps subsequent records to that schema. `LENIENT` mode does not reorder the sink columns; record values are mapped to the unchanged column order. Explicitly excluding `reorder.column` also keeps the sink column order unchanged.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: There are three documentation gaps:

  1. Hudi is missing from the list of sinks that reject reorder explicitly; HudiMetadataApplier currently fails TRY_EVOLVE as described in the review comment on SchemaChangeEventVisitor.
  2. The TRY_EVOLVE sentence is only true for sinks that throw UnsupportedSchemaChangeEventException, but StarRocks validation, Paimon missing columns and SchemaUtils currently throw other exceptions. Either fix those or qualify the sentence.
  3. The StarRocks connector page (docs/content/docs/connectors/pipeline-connectors/starrocks.md:257-261) still says only add/drop/rename/alter-type and "new column always last". Document reorder support, the key-order constraint, and the one-schema-change-at-a-time behavior.

Suggestion: Also consider an upgrade note: the default accepted set is ALL (SinkDef.java:56-57), so existing EVOLVE pipelines will now fail or physically reorder on the next MODIFY ... AFTER.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants