Repository navigation
[FLINK-36263][pipeline-connector/kafka] Include only changed fields in Canal JSON old - #4603
Open
Pika-Yalei wants to merge 2 commits into
Open
Pika-Yalei wants to merge 2 commits into
Pika-Yalei wants to merge 2 commits into
Conversation
…n Canal JSON old Filter unchanged fields from the old row using Flink's native JSON converters, while preserving complete data rows and existing JSON format options. Add regression coverage for null transitions, complex values, repeated events, and schema changes. Generated-by: OpenAI Codex (model version not exposed)
…and simplify test imports Move the row serializer into a package-private top-level class without changing serialization behavior. Use simple type names and explicit imports in tests. Generated-by: OpenAI Codex (model version not exposed)
Pika-Yalei
marked this pull request as ready for review
October 27, 2026 02:01
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of this pull request?
Fix FLINK-36263: Canal JSON UPDATE messages currently include every column in
old, rather than only the columns whose values changed.Brief change log
oldusing structural JSON equality.datarow, configured null-field handling, and existing timestamp, map-key and decimal serialization options.Verifying this change
The updated Canal serializer tests failed against the original implementation (5 tests, 5 assertion failures, 0 errors), then passed with the fix.
All seven Kafka connector unit-test classes passed in each configuration (24 tests, 0 failures, 0 errors, 0 skipped):
testclean test-Pflink2 clean packageCommands use:
mvn -pl :flink-cdc-pipeline-connector-kafka -am \ -Dtest=JsonSerializationSchemaTest,CsvSerializationSchemaTest,KafkaDataSinkFactoryTest,RecordDataConverterTest,TableSchemaInfoTest,CanalJsonSerializationSchemaTest,DebeziumJsonSerializationSchemaTest \ -Dsurefire.failIfNoSpecifiedTests=false testFor the JDK 17 runs, replace the final goal with
clean testor-Pflink2 clean package, respectively. The Flink 2 profile requires packaging the compatibility artifact before the downstream dependency-copy execution.Spotless formatting/checks, Checkstyle, and
git diff --checkpassed. Docker-based integration/end-to-end tests have not been run (Docker is unavailable in the development environment). Upstream CI remains to be verified; this PR is intentionally a draft.Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: OpenAI Codex (model version not exposed)