Repository navigation
Conversation
Port column reorder events and adapt schema visitors, serializers, synchronous and asynchronous transforms, and connector handling to the upstream codebase. Preserve existing event identifiers and legacy add-column serialization. Generated-by: Codex CLI 0.163.0-alpha.10.2 AI-Contributed/Feature: 1583/1583 AI-Contributed/UT: 3758/3758
Use same-type name/city columns to verify that reordering preserves their association with values, instead of merely catching incompatible layouts. Generated-by: Codex CLI 0.163.0-alpha.10.2 AI-Contributed/Feature: 0/0 AI-Contributed/UT: 24/24
|
@sd4324530 CC as you may be interested in this. |
Use fragment matching while waiting for initial table schemas, and align Kafka Debezium JSON expectations with upstream source metadata. Generated-by: Codex CLI 0.163.0-alpha.10.2 AI-Contributed/Feature: 0/0 AI-Contributed/UT: 16/16
sd4324530
left a comment
There was a problem hiding this comment.
Thanks for the work on FLINK-41485. The overall approach is solid: a validated, immutable ReorderColumnEvent, transform-side move derivation, and per-sink handling with documentation and tests is the right direction, and it fixes the basic MySQL → Paimon/Kafka misalignment case.
I left 14 inline comments. Before merging, I would like the following blockers addressed:
- Non-asterisk transforms still silently misalign. The reorder event is dropped in the non-asterisk pre-transform path while
PreTransformOperatorrebuilds its processor from the reordered source schema, so downstream rows change order without a schema update. Please either forward a derived pre-transform reorder event or keep the output order stable, and add a non-asterisk projection test. - StarRocks
MODIFY ... AFTERfails deterministically. TheAlterColumnTypeEventstarts an asynchronous StarRocks schema-change job, and the immediately following reorderALTERis rejected while the table is inSCHEMA_CHANGE. Please wait for the type job (likeexecuteAlter) or combine type and position into oneALTER, and add a type + reorder integration test. - Hudi cannot tolerate reorder in
TRY_EVOLVE. The 8-arg visitor default throwsUnsupportedSchemaChangeEventException, butHudiMetadataApplierwraps it in a plainRuntimeException, so the coordinator does not ignore it. Please use the 10-arg overload, do not wrapSchemaEvolveException, and add Hudi to the documentation. - Excluding column changes can break reorder. With
exclude.schema.changes: [add.column](orrename.column) butreorder.columnincluded, the evolved schema lacks the referenced column and the job fails even inTRY_EVOLVE. Please filter/rewrite moves against the evolved schema, or throwUnsupportedSchemaChangeEventException, and cover this configuration in a test.
The remaining inline comments are smaller suggestions: StarRocks job amplification and connection/retry handling, unsupported-move exception types, the information_schema column-order assumption, MySQL identifier casing, LCS memory, duplicated move logic, test coverage for metadata preservation, and documentation/upgrade notes.
Disclosure: this review was primarily generated by an AI assistant (Claude) from a static review of the PR diff, and was reviewed and confirmed by me before posting. I did not run the new unit/integration tests locally, so the transform and StarRocks scenarios in particular should be validated with targeted tests.
| return position; | ||
| } | ||
|
|
||
| public org.apache.flink.cdc.common.event.ColumnPosition getColumnPosition() { |
There was a problem hiding this comment.
Problem: getColumnPosition() dereferences position without a null check, but the still-public deprecated constructor at line 130 accepts a null position. Code that previously relied on the null-safe AddColumnEvent.ColumnPosition.LAST.equals(e.getPosition()) check in SchemaUtils.transformSchemaChangeEvent will now throw an NPE.
Suggestion: Make this method null-safe (return null, or checkNotNull with a clear message), or validate position in the deprecated constructor. Low likelihood, but it is a behavior change.
|
|
||
| if (hasAsteriskMap.getOrDefault(tableId, true)) { | ||
| // See comments in PreTransformOperator#cacheChangeSchema method. | ||
| if (event instanceof ReorderColumnEvent) { |
There was a problem hiding this comment.
Problem: Non-asterisk projections still silently misalign. When hasAsteriskMap is false, PreTransformOperator.cacheChangeSchema calls SchemaUtils.transformSchemaChangeEvent(false, ...), which returns Optional.empty() for ReorderColumnEvent (SchemaUtils.java:379-383), so the event never reaches PostTransformOperator. However, cachePreTransformProcessor (PreTransformOperator.java:259) rebuilds the pre-transform processor from the reordered source schema, so the rows emitted by PreTransformOperator change order while this operator pre-schema stays the same. Every subsequent record is then decoded positionally with the old schema, and values are silently swapped.
Suggestion: Handle reorder in the non-asterisk branch as well — for example, after rebuilding the processor, derive oldPreSchema/newPreSchema and forward ReorderColumnEventTransformer.transformReorderColumnEvent(...), or keep the pre-transform output order stable. Add a test with a non-asterisk projection such as id, age and a move of age; the current test only covers * (testReorderWithPreWildcardAndComputedColumn).
| } | ||
| } | ||
|
|
||
| private void applyReorderColumn(ReorderColumnEvent reorderColumnEvent) |
There was a problem hiding this comment.
Problem: MySQL parses MODIFY COLUMN ... AFTER into an AlterColumnTypeEvent followed by this ReorderColumnEvent (CustomAlterTableParserListener.java:450-471). applyAlterColumnType (line 324) calls StarRocksEnrichedCatalog.alterColumnType, which submits an asynchronous StarRocks schema-change job and returns without waiting (StarRocksEnrichedCatalog.java:121-152). This method then immediately issues the position ALTER. StarRocks allows only one schema change per table, so the second ALTER is rejected with SCHEMA_CHANGE, and the job fails deterministically. This also happens for a pure position change, because the type event is always emitted.
Suggestion: Make alterColumnType wait for its job like executeAlter, wait for any in-flight column alter job before the first move, or combine type and position into one ALTER. Add a test that applies AlterColumnTypeEvent and then ReorderColumnEvent; the current IT only sends the reorder.
| renameColumnEventVisitor, | ||
| truncateTableEventVisitor, | ||
| alterTableCommentEventVisitor, | ||
| reorderColumnEvent -> { |
There was a problem hiding this comment.
Problem: The 8-argument overloads now throw UnsupportedSchemaChangeEventException for reorder, but the only in-tree caller, HudiMetadataApplier (HudiMetadataApplier.java:123-160), catches Exception and wraps it in a plain RuntimeException. SchemaCoordinator.shouldIgnoreException only tolerates UnsupportedSchemaChangeEventException in TRY_EVOLVE, so Hudi jobs fail instead of keeping the sink schema. Hudi is also missing from the new documentation list.
Suggestion:
- Update Hudi to use the 10-arg overload with an explicit
reorder -> throw new UnsupportedSchemaChangeEventException(event)handler, and do not wrapSchemaEvolveException. - Add Hudi to the docs.
- Consider deprecating the 8-arg overloads so third-party sinks migrate.
| return candidate; | ||
| } | ||
| } | ||
| throw new IllegalArgumentException( |
There was a problem hiding this comment.
Problem: This throws for a moved column (line 253 does the same for a missing anchor) that is absent from the evolved schema. That happens for a supported configuration: exclude add.column (or rename.column) but keep reorder.column. The original schema has the column, the evolved schema does not, and applying the reorder fails here. SchemaCoordinator.shouldIgnoreException does not tolerate IllegalArgumentException, so even TRY_EVOLVE fails. The new E2E testExcludedReorderAndAddColumns excludes both types, so it does not cover this.
Suggestion: Rewrite/filter reorder moves against the current evolved schema in SchemaDerivator.normalizeSchemaChangeEvents (drop moves whose moved or reference column is absent, and drop the event if nothing remains), or throw UnsupportedSchemaChangeEventException. Add a test that excludes only add.column.
|
|
||
| private static List<String> longestCommonSubsequence( | ||
| List<String> oldColumnNames, List<String> newColumnNames) { | ||
| int[][] lengths = new int[oldColumnNames.size() + 1][newColumnNames.size() + 1]; |
There was a problem hiding this comment.
Problem: This allocates an (n+1) x (m+1) int matrix per transform instance per reorder event. For a 10,000-column table that is about 400 MB and about 100M comparisons; 1,000 columns is about 4 MB per instance.
Suggestion: Since the two schemas are permutations of the same column set, compute the LIS over the old-index to new-index permutation in O(n log n), or derive moves directly from the event. At minimum, guard the matrix size.
| newColumnPositions.put(newColumnNames.get(i), i); | ||
| } | ||
| List<ReorderColumnEvent.ColumnMove> columnMoves = new ArrayList<>(); | ||
| if (stableAnchors.isEmpty()) { |
There was a problem hiding this comment.
Problem: This branch is unreachable: two non-empty permutations of the same column set always have an LCS of at least one element, and equal lists return at line 45. It has no test coverage.
Suggestion: Remove it. The targetIndex = 0 initialization in the following loop already handles the first-anchor case.
| private boolean applyColumnMove( | ||
| List<String> simulatedColumnNames, ReorderColumnEvent.ColumnMove columnMove) { | ||
| List<String> reorderedColumnNames = new ArrayList<>(simulatedColumnNames); | ||
| checkArgument( |
There was a problem hiding this comment.
Problem: Move application is now implemented four times with slightly different semantics: SchemaUtils.applyReorderColumnEvent (SchemaUtils.java:192-259), this method, StarRocksMetadataApplier.applyColumnMoves (lines 438-491), and ReorderColumnEventTransformer. Missing columns here throw IllegalArgumentException, which TRY_EVOLVE does not tolerate.
Suggestion: Extract a shared helper (for example applyMoves(List<String>, List<ColumnMove>) returning the reordered names, or a normalized/minimal move list on ReorderColumnEvent) so no-op handling and missing-column policy stay consistent. Use UnsupportedSchemaChangeEventException for unsupported moves.
| } | ||
|
|
||
| @Test | ||
| void testReorderColumnSupportsMovesAndPreservesMetadata() throws Exception { |
There was a problem hiding this comment.
Problem: This test cannot catch metadata loss: MockStarRocksCatalog.alterColumnPosition (MockStarRocksCatalog.java:200) ignores the passed StarRocksColumn/isKey and just reorders the existing column objects, so dropping DEFAULT/nullability/COMMENT in loadColumnDefault/buildAlterColumnPositionSql leaves it green. The IT (StarRocksMetadataApplierITCase.java:737) also sends only the reorder event and has no defaults/comments.
Suggestion: Make the mock apply the restated definition (or assert on real information_schema values), and add an IT with DEFAULT/COMMENT and a preceding AlterColumnTypeEvent. Also cover defaults like CURRENT_TIMESTAMP(3), which formatDefaultValue currently quotes as a string.
|
|
||
| The MySQL pipeline source emits `ReorderColumnEvent` for `MODIFY COLUMN` and `CHANGE COLUMN` statements with `FIRST` or `AFTER` clauses. Renaming, type changes, and column moves in a compound DDL are applied in statement order. The `reorder` and `column` families both include `reorder.column`. | ||
|
|
||
| Kafka updates its serialization schema. Paimon applies physical column moves, while StarRocks only accepts moves that keep primary key columns first and in their existing order. Doris, Fluss, Iceberg, MaxCompute, and OceanBase reject physical column reordering explicitly. In `EVOLVE` mode, an unsupported move fails the job rather than silently misaligning field values. In `TRY_EVOLVE` mode, the framework retains the sink schema and maps subsequent records to that schema. `LENIENT` mode does not reorder the sink columns; record values are mapped to the unchanged column order. Explicitly excluding `reorder.column` also keeps the sink column order unchanged. |
There was a problem hiding this comment.
Problem: There are three documentation gaps:
- Hudi is missing from the list of sinks that reject reorder explicitly;
HudiMetadataAppliercurrently failsTRY_EVOLVEas described in the review comment onSchemaChangeEventVisitor. - The
TRY_EVOLVEsentence is only true for sinks that throwUnsupportedSchemaChangeEventException, but StarRocks validation, Paimon missing columns andSchemaUtilscurrently throw other exceptions. Either fix those or qualify the sentence. - The StarRocks connector page (
docs/content/docs/connectors/pipeline-connectors/starrocks.md:257-261) still says only add/drop/rename/alter-type and "new column always last". Document reorder support, the key-order constraint, and the one-schema-change-at-a-time behavior.
Suggestion: Also consider an upgrade note: the default accepted set is ALL (SinkDef.java:56-57), so existing EVOLVE pipelines will now fail or physically reorder on the next MODIFY ... AFTER.
What is the purpose of this pull request?
Fix FLINK-41485: propagate MySQL column reordering so downstream schemas stay aligned with the layout of subsequent records. Ignoring
MODIFY COLUMN/CHANGE COLUMNwithFIRSTorAFTERcan otherwise silently assign values to the wrong fields.JIRA: https://issues.apache.org/jira/browse/FLINK-41485
Brief change log
Verifying this change
name/cityfields retain their values after reordering.Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: Codex CLI 0.163.0-alpha.10.2