Skip to content

cdc: add avro-include-before-value support - #22015

Open
wk989898 wants to merge 1 commit into
pingcap:masterfrom
wk989898:avro-before
Open

wk989898 wants to merge 1 commit into
pingcap:masterfrom
wk989898:avro-before

Conversation

@wk989898

@wk989898 wk989898 commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

First-time contributors' checklist

What is changed, added, or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

  • master (the latest development version)
  • v9.0 (TiDB 9.0 versions)
  • v8.5 (TiDB 8.5 versions)
  • v8.1 (TiDB 8.1 versions)
  • v7.5 (TiDB 7.5 versions)
  • v7.1 (TiDB 7.1 versions)
  • v6.5 (TiDB 6.5 versions)

What is the related PR or file link(s)?

  • Related code change PR links (if applicable):
  • This PR is translated from:
  • Other reference link(s):

AI agent involvement

  • The changes in this PR were primarily made by an AI agent on behalf of the PR author.

Do your changes match any of the following descriptions?

  • Delete files
  • Change aliases
  • Need modification after applied to another branch
  • Might cause conflicts after applied to another branch

Summary by CodeRabbit

  • 文档
    • 补充 Kafka Avro 协议的旧值输出配置说明:开启后,消息可携带更改前的行数据,Delete 事件也会包含 Value,并通过 _tidb_op 标识事件类型。
    • 说明该配置默认关闭,仅适用于 TiCDC 新架构及 Avro 协议,并明确其对校验和计算的影响。

Signed-off-by: wk989898 <nhsmwk@gmail.com>
@ti-chi-bot

ti-chi-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign overvenus for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added missing-translation-status This PR does not have translation status info. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 29, 2026
@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Walkthrough

本次更新 TiCDC 文档,说明 avro-include-before-value 的配置方式、适用条件和启用后的 Avro 事件格式,并更新消费者事件识别与 Delete Checksum 校验说明。

Changes: Avro 旧值配置与事件说明

Layer / File(s) Summary
配置入口与适用范围
ticdc/ticdc-changefeed-config.md, ticdc/ticdc-sink-to-kafka.md
新增 avro-include-before-value 的配置说明,包括默认值、适用的 TiCDC 架构和 Avro 协议,以及启用后的旧值字段和 Delete 标识。调整 Simple Protocol 说明。
Avro 事件格式与校验规则
ticdc/ticdc-avro-protocol.md, ticdc/ticdc-integrity-check.md
说明默认不输出旧值,以及启用选项后 _ticdc_before、_tidb_op 和事件识别规则的变化。补充启用选项后的 Delete Checksum 计算说明。

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Suggested reviewers: 3aceshowhand, oreoxmt

Merge Risk: Moderate · up to 13b0f

The new option's documentation conflicts with existing statements. The checksum verification example may fail for users who enable avro-include-before-value. Update the example and fix the stale Delete output wording before merging.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Description check Warning 描述包含 CLA、适用版本和其他模板检查项,但“ What is changed, added, or deleted? ”这一必填部分为空,未说明本次文档变更内容和原因。 在“ What is changed, added, or deleted? ”部分补充变更说明。请说明新增 avro-include-before-value 配置及其在 Avro、Kafka Sink 和校验文档中的行为说明,并说明相关原因。
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed 标题明确说明了为 CDC 添加 avro-include-before-value 支持,能够概括本次文档变更的主要主题。
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (4 skipped: 4 unsupported.)

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3


Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 9f466f0 and 13b0f2e.

Files selected for processing: 4
  • ticdc/ticdc-avro-protocol.md
  • ticdc/ticdc-changefeed-config.md
  • ticdc/ticdc-integrity-check.md
  • ticdc/ticdc-sink-to-kafka.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Comment on lines +104 to +112
开启后,Key 的编码方式不变,Value 中新增 `_ticdc_before` 字段。不同类型的 DML 事件编码如下:

| 事件类型 | Value 顶层列字段 | `_ticdc_before` | `_tidb_op` |
| :------- | :--------------- | :-------------- | :--------- |
| Insert | 插入后的行数据 | `null` | `c` |
| Update | 更新后的行数据 | 更新前的行数据 | `u` |
| Delete | 删除前的行数据 | 删除前的行数据 | `d` |

`_ticdc_before` 的 Avro 类型为 `null` 与嵌套 `record` 的联合类型,默认值为 `null`。嵌套 `record` 中的列定义与 Value 顶层的列定义相同。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Stability & Availability · Quick win

Supported by static analysis

Script executed:

ls ticdc | rg -i 'avro'
rg -n '_tidb_op|_ticdc_before|type assertion|\.\(' ticdc/ticdc-avro-checksum-verification.md
sed -n '76,135p' ticdc/ticdc-avro-protocol.md

Repository: pingcap/docs-cn

Length of output: 3939


Script executed:

sed -n '225,290p' ticdc/ticdc-avro-checksum-verification.md
rg -n -F -e 'avro-include-before-value' -e '_ticdc_before' -e '校验' -e 'checksum' -e 'Checksum' -- ticdc/ticdc-avro-checksum-verification.md ticdc/ticdc-avro-protocol.md || test "$?" -eq 1

Repository: pingcap/docs-cn

Length of output: 9087


同步更新 Checksum 校验示例,跳过 _ticdc_before。

开启 avro-include-before-value 后,若 _ticdc_before 位于 _tidb_op 之前,ticdc/ticdc-avro-checksum-verification.md 中的示例会失败。CalculateAndVerifyChecksum() 只在遇到 _tidb_op 时停止遍历,并将之前的字段按普通列处理。处理 _ticdc_before 时,代码进入 []interface{} 联合类型分支,并对 m["connect.parameters"] 直接执行断言。若嵌套 record 未提供该字段,断言会 panic。若提供了该字段,代码会把嵌套 record 的类型信息当作普通列处理,同样会失败。该错误会返回给调用方,调用方随后执行 log.Panic。

请在遍历 Value 字段时跳过 _ticdc_before,只对 Value 顶层真实列计算 Checksum。同时更新示例对应的测试。

🐛 建议修改
         colName := field["name"].(string)
+        if colName == "_ticdc_before" {
+            continue
+        }
         if colName == "_tidb_op" {
             break
         }

Comment on lines +122 to +124
默认情况下,Avro 只编码在 DML 事件中发生数据变更的行的所有列数据信息,不收集数据变更的类型和 TiDB 专有的 CommitTS 事务唯一标识信息。为了解决这个问题,TiCDC 在 Avro 协议格式中附加了 TiDB 扩展字段。当 `sink-uri` 中设置 `enable-tidb-extension` 为 `true` (默认为 `false`)后,TiCDC 生成 Avro 消息时,Value 部分会包含以下三个字段:

- `_tidb_op`:DML 的类型,"c" 表示插入,"u" 表示更新。
- `_tidb_op`:DML 的类型,`c` 表示插入,`u` 表示更新,`d` 表示删除。Delete 事件仅在开启 `avro-include-before-value` 时包含该字段。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Data Integrity & Integration · Quick win

限定 TiDB 扩展字段对 Delete 的适用范围。

当前文字说明 Value 会包含三个扩展字段,但前文已说明未开启 avro-include-before-value 时 Delete 的 Value 为空。此时三个字段都不会出现在 Delete 事件中。当前只对 _tidb_op 添加例外,会误导消费者读取不存在的字段。(raw.githubusercontent.com)

可直接应用的替换
-默认情况下,Avro 只编码在 DML 事件中发生数据变更的行的所有列数据信息,不收集数据变更的类型和 TiDB 专有的 CommitTS 事务唯一标识信息。为了解决这个问题,TiCDC 在 Avro 协议格式中附加了 TiDB 扩展字段。当 `sink-uri` 中设置 `enable-tidb-extension` 为 `true` (默认为 `false`)后,TiCDC 生成 Avro 消息时,Value 部分会包含以下三个字段:
+默认情况下,Avro 只编码在 DML 事件中发生数据变更的行的所有列数据信息,不收集数据变更的类型和 TiDB 专有的 CommitTS 事务唯一标识信息。为了解决这个问题,TiCDC 在 Avro 协议格式中附加了 TiDB 扩展字段。当 `sink-uri` 中设置 `enable-tidb-extension` 为 `true`(默认为 `false`)后,TiCDC 生成 Avro 消息时,Insert 和 Update 事件的 Value 部分会包含以下三个字段;Delete 事件仅在同时开启 `avro-include-before-value` 时包含这些字段:
-- `_tidb_op`:DML 的类型,`c` 表示插入,`u` 表示更新,`d` 表示删除。Delete 事件仅在开启 `avro-include-before-value` 时包含该字段。
+- `_tidb_op`:DML 的类型,`c` 表示插入,`u` 表示更新,`d` 表示删除。

##### `avro-include-before-value` <span class="version-mark">从 v8.5.9 版本开始引入</span>

- 是否在 Avro 消息中输出行数据更改前的值。该参数仅适用于 [TiCDC 新架构](/ticdc/ticdc-architecture.md),且仅在输出协议为 `avro` 时生效。
- 开启后,Value 中新增 `_ticdc_before` 字段,Update 和 Delete 事件通过该字段输出旧值,Insert 事件中该字段为 `null`。Delete 事件也包含 Value,并通过 `_tidb_op=d` 标识。详情请参考[输出旧值](/ticdc/ticdc-avro-protocol.md#输出旧值)。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Data Integrity & Integration · Quick win

Supported by static analysis

Script executed:

rg -n 'delete-only-output-handle-key-columns|avro-include-before-value' ticdc/ticdc-changefeed-config.md
sed -n '295,345p' ticdc/ticdc-changefeed-config.md

Repository: pingcap/docs-cn

Length of output: 1905


Script executed:

nl -ba ticdc/ticdc-changefeed-config.md | sed -n '195,210p'

Repository: pingcap/docs-cn

Length of output: 849


同步修正 Avro Delete 输出说明。

启用 avro-include-before-value 后,Delete 事件的 Value 会包含删除前的行数据。第 206 行仍说明 Avro Delete 事件总是只输出主键列或唯一索引列,与第 335 行冲突,可能误导下游 Schema 和消费者实现。

建议将第 206 行替换为:

Avro 协议不受该参数控制。默认情况下,Delete 事件只输出 Key(主键列或唯一索引列);启用 avro-include-before-value 后,Delete 事件的 Value 还会包含删除前的行数据。

@qiancai qiancai added the v9.0-beta.3 This PR/issue applies to TiDB v9.0-beta.3. label Oct 29, 2026
@ti-chi-bot

ti-chi-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

@ginkgoch: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

missing-translation-status This PR does not have translation status info. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. v9.0-beta.3 This PR/issue applies to TiDB v9.0-beta.3.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants