Skip to content

[FLINK-36263][pipeline-connector/kafka] Include only changed fields in Canal JSON old - #4603

Open
Pika-Yalei wants to merge 2 commits into
apache:masterfrom
Pika-Yalei:flink-36263-canal-old
Open

Pika-Yalei wants to merge 2 commits into
apache:masterfrom
Pika-Yalei:flink-36263-canal-old

Conversation

@Pika-Yalei

@Pika-Yalei Pika-Yalei commented Oct 27, 2026 •

Copy link
Copy Markdown

What is the purpose of this pull request?

Fix FLINK-36263: Canal JSON UPDATE messages currently include every column in old, rather than only the columns whose values changed.

Brief change log

  • Build the Canal JSON envelope with Flink's native row-to-JSON converters and remove unchanged fields from old using structural JSON equality.
  • Preserve the complete data row, configured null-field handling, and existing timestamp, map-key and decimal serialization options.
  • Keep INSERT and DELETE behavior unchanged.
  • Keep the row serializer in a package-private top-level class and use explicit imports in the tests; no public API is added.
  • Add regression coverage for null-to-value/value-to-null changes, entirely unchanged rows, multiple changed complex values, independently constructed equal values and map ordering, consecutive events, schema additions, and REPLACE events with a before image.

Verifying this change

The updated Canal serializer tests failed against the original implementation (5 tests, 5 assertion failures, 0 errors), then passed with the fix.

All seven Kafka connector unit-test classes passed in each configuration (24 tests, 0 failures, 0 errors, 0 skipped):

  • JDK 11 / Flink 1.20.3: test
  • JDK 17 / Flink 1.20.3: clean test
  • JDK 17 / Flink 2.2.0: -Pflink2 clean package

Commands use:

mvn -pl :flink-cdc-pipeline-connector-kafka -am \
  -Dtest=JsonSerializationSchemaTest,CsvSerializationSchemaTest,KafkaDataSinkFactoryTest,RecordDataConverterTest,TableSchemaInfoTest,CanalJsonSerializationSchemaTest,DebeziumJsonSerializationSchemaTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

For the JDK 17 runs, replace the final goal with clean test or -Pflink2 clean package, respectively. The Flink 2 profile requires packaging the compatibility artifact before the downstream dependency-copy execution.

Spotless formatting/checks, Checkstyle, and git diff --check passed. Docker-based integration/end-to-end tests have not been run (Docker is unavailable in the development environment). Upstream CI remains to be verified; this PR is intentionally a draft.

Documentation

  • Does this pull request introduce a new feature? No; it corrects the existing Canal JSON UPDATE representation.
  • If yes, how is the feature documented? Not applicable.

Was generative AI tooling used to co-author this PR?
  • Yes — OpenAI Codex. The exact model version is not exposed by the execution environment.

Generated-by: OpenAI Codex (model version not exposed)

…n Canal JSON old

Filter unchanged fields from the old row using Flink's native JSON converters,
while preserving complete data rows and existing JSON format options.
Add regression coverage for null transitions, complex values, repeated events,
and schema changes.

Generated-by: OpenAI Codex (model version not exposed)
…and simplify test imports

Move the row serializer into a package-private top-level class without changing
serialization behavior. Use simple type names and explicit imports in tests.

Generated-by: OpenAI Codex (model version not exposed)
@Pika-Yalei
Pika-Yalei marked this pull request as ready for review October 27, 2026 02:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant