cdc: update storage sink by wk989898 · Pull Request #21221 · pingcap/docs-cn

wk989898 · 2025-12-26T05:56:53Z

First-time contributors' checklist

I've signed the Contributor License Agreement, which is required for the repository owners to accept my contribution.

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s): sink: add dispatcherID in data file name for storage sink ticdc#3664 sink: modify the write order of the index file and data file in storage writer ticdc#3785

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

wk989898 · 2025-12-26T05:59:29Z

+
+在处理文件时，可能会遇到某个文件还没有写入完成就进行读取的情况，这会导致部分数据没有成功读取。我们需要先读取 Index 文件来获取可以进行处理的文件来避免这种情况发生。消费逻辑为：
+- 读取目录下的 meta/CDC.index 文件，获取当前已经完成写入的文件名。
+- 依次处理文件序号小于等于该文件名的 DML 事件。


在新架构上为小于等于，在老架构上应该为小于

@qiancai 新老架构逻辑基本一致，只是在处理文件时，老架构只能处理小于 index 文件中的文件序号的 dml 文件。新架构可以处理当前文件序号的 dml 文件

3AceShowHand · 2025-12-26T06:57:07Z

+新架构支持为 MySQL Sink 和 Storage Sink 启用**表级任务拆分**。你可以通过在 Changefeed 配置中设置 `scheduler.enable-table-across-nodes = true` 来启用该功能。

-启用后，当**有且仅有一个主键或非空唯一键**的表满足以下任一条件时，TiCDC 会自动将其拆分并分发到多个节点并行执行同步，从而提升同步效率与资源利用率：
+对于 MySQL Sink 来说，**有且仅有一个主键或非空唯一键**的表满足以下任一条件时，TiCDC 会自动将其拆分并分发到多个节点并行执行同步，从而提升同步效率与资源利用率：


添加高亮说明。

ti-chi-bot · 2025-12-26T06:59:08Z

@3AceShowHand: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hongyunyan · 2025-12-29T03:05:04Z

+新架构支持为 MySQL Sink 和 Storage Sink 启用**表级任务拆分**。你可以通过在 Changefeed 配置中设置 `scheduler.enable-table-across-nodes = true` 来启用该功能。

-启用后，当**有且仅有一个主键或非空唯一键**的表满足以下任一条件时，TiCDC 会自动将其拆分并分发到多个节点并行执行同步，从而提升同步效率与资源利用率：
+对于 **MySQL Sink** 来说，**有且仅有一个主键或非空唯一键**的表满足以下任一条件时，TiCDC 会自动将其拆分并分发到多个节点并行执行同步，从而提升同步效率与资源利用率：


这里感觉要找找老板确认一下，我们对于这个拆分条件要说到什么程度。目前是默认 enable-splittable-check 为 false，但是 mysql sink 时，默认 enable-splittable-check 为 true。只有 enable-splittable-check 为 true 的时候我们会校验 “有且仅有一个主键或非空唯一键”。对于 mysql 我们又可以通过 force-split 强制开启拆分（不过 force split 估计不用暴露给用户了）

ti-chi-bot · 2026-02-25T06:24:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from qiancai. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Co-authored-by: Grace Cai <qqzczy@126.com>

ti-chi-bot · 2026-03-02T03:44:08Z

@wk989898: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-verify	`fa8e58b`	link	true	`/test pull-verify`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

qiancai · 2026-03-27T08:37:08Z

According to @wk989898, the changes of this doc PR are not planned in v8.5.6, so we'll hold this PR for now.

Synced from: pingcap/docs-cn#21221 Target PR: pingcap#22738 AI Provider: azure Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

cdc: update storage sink

d89d945

ti-chi-bot Bot added missing-translation-status This PR does not have translation status info. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Dec 26, 2025

wk989898 commented Dec 26, 2025

View reviewed changes

wk989898 added 2 commits December 26, 2025 06:00

chore

f828645

update

e391a99

ti-chi-bot Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Dec 26, 2025

chore

66a41c8

3AceShowHand reviewed Dec 26, 2025

View reviewed changes

3AceShowHand approved these changes Dec 26, 2025

View reviewed changes

Update ticdc-architecture.md

fc88d2b

hongyunyan reviewed Dec 29, 2025

View reviewed changes

qiancai self-assigned this Jan 22, 2026

qiancai added area/ticdc Indicates that the Issue or PR belongs to the area of TiCDC. translation/doing This PR’s assignee is translating this PR. labels Jan 22, 2026

ti-chi-bot Bot removed the missing-translation-status This PR does not have translation status info. label Jan 22, 2026

qiancai added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Jan 22, 2026

Merge branch 'master' into storage-newarch

df22a64

qiancai reviewed Feb 26, 2026

View reviewed changes

Comment thread ticdc/ticdc-architecture.md Outdated

Comment thread ticdc/ticdc-architecture.md

Comment thread ticdc/ticdc-architecture.md

Comment thread ticdc/ticdc-architecture.md Outdated

qiancai reviewed Feb 26, 2026

View reviewed changes

Comment thread ticdc/ticdc-architecture.md Outdated

qiancai reviewed Feb 26, 2026

View reviewed changes

Comment thread ticdc/ticdc-storage-consumer-dev-guide.md Outdated

qiancai reviewed Feb 26, 2026

View reviewed changes

Comment thread ticdc/ticdc-architecture.md Outdated

wk989898 and others added 4 commits February 27, 2026 15:30

Apply suggestions from code review

f0cd2b7

Co-authored-by: Grace Cai <qqzczy@126.com>

Update ticdc-architecture.md

725a07b

Update ticdc-architecture.md

a5fff46

Update ticdc-storage-consumer-dev-guide.md

6375a32

qiancai reviewed Feb 27, 2026

View reviewed changes

wk989898 and others added 2 commits February 27, 2026 17:00

Apply suggestions from code review

01d4504

Co-authored-by: Grace Cai <qqzczy@126.com>

Apply suggestion from @qiancai

e03fa03

Co-authored-by: Grace Cai <qqzczy@126.com>

qiancai reviewed Feb 27, 2026

View reviewed changes

Comment thread ticdc/ticdc-architecture.md

wk989898 and others added 2 commits February 27, 2026 17:20

Apply suggestion from @qiancai

fa01432

Co-authored-by: Grace Cai <qqzczy@126.com>

Update ticdc-storage-consumer-dev-guide.md

27ec9fd

qiancai reviewed Feb 27, 2026

View reviewed changes

Comment thread ticdc/ticdc-architecture.md

qiancai reviewed Feb 27, 2026

View reviewed changes

Comment thread ticdc/ticdc-storage-consumer-dev-guide.md Outdated

Apply suggestions from code review

e19e96c

Co-authored-by: Grace Cai <qqzczy@126.com>

qiancai added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 27, 2026

qiancai reviewed Mar 2, 2026

View reviewed changes

Comment thread ticdc/ticdc-storage-consumer-dev-guide.md Outdated

fix broken links

fa8e58b

qiancai mentioned this pull request Apr 13, 2026

cdc: update storage sink pingcap/docs#22738

Open

14 tasks

Conversation

wk989898 commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

First-time contributors' checklist

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

Uh oh!

wk989898 Dec 26, 2025

Choose a reason for hiding this comment

Uh oh!

wk989898 Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

3AceShowHand Dec 26, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot Bot commented Dec 26, 2025

Uh oh!

hongyunyan Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot Bot commented Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ti-chi-bot Bot commented Mar 2, 2026

Uh oh!

qiancai commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wk989898 commented Dec 26, 2025 •

edited

Loading