[BugFix] Fix KSM overlap bug#7772
Conversation
|
Thanks for your contribution! |
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-11 15:36:26
📋 Review 摘要
PR 概述：修复 KSM overlap 模式下 accept_num_cpu 同步问题导致 logprobs 截断不正确的 bug
变更范围：fastdeploy/model_executor/pre_and_post_process.py（save_output_speculate 函数）
影响面 Tag：[Speculative Decoding] [BugFix]
📝 PR 规范检查
存在以下规范问题：
- 标题缺少官方 Tag：当前标题
fix ksm overlap bugs未包含任何官方 Tag。 - 目标分支为
release/2.6：需使用 Cherry-Pick 格式，并确认原 develop 分支 PR 编号。
标题建议（可直接复制）：
[Cherry-Pick][BugFix][Speculative Decoding] Fix accept_num_cpu sync bug when KSM overlap is enabled (#<原PR号>)
PR 描述建议（可直接复制）：
## Motivation
修复 KSM overlap 模式下 logprobs 重归一化逻辑中 `accept_num_cpu` 同步问题。当 overlap 启用时，`accept_num_cpu`（D2H 异步拷贝）可能尚未更新，导致 `real_token_num` 取值错误，进而引发 logprobs 截断不正确。直接使用 GPU 侧的 `accept_num` 可规避该竞态。
## Modifications
- `fastdeploy/model_executor/pre_and_post_process.py`：在 `save_output_speculate` 函数的 logprobs 重归一化路径中，将 `share_inputs["accept_num_cpu"].sum()` 替换为 `share_inputs["accept_num"].sum()`，并移除相关 TODO 注释。
## Usage or Command
N/A
## Accuracy Tests
N/A
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [x] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [ ] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 📝 PR 规范 | — | 标题缺少官方 Tag，目标 release 分支未使用 Cherry-Pick 格式，描述所有章节均为空占位符 |
| ❓ 疑问 | pre_and_post_process.py:588 |
share_inputs["accept_num"].sum() 若返回 GPU 0-dim tensor，用作切片索引时会触发隐式 D2H 同步；建议确认是否需要 .item() |
说明：疑问项行内评论因 diff 解析限制未能附加到具体行，请作者参考以上说明确认。
总体评价
代码改动精准，将 overlap 模式下可能过时的 CPU 侧拷贝 accept_num_cpu 替换为直接读取 GPU 侧的 accept_num，修复方向正确；无 P0 阻塞性问题，需补全 PR 标题格式与描述内容后即可合入。
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览当前有 1 个 Required 任务失败（
2 任务状态汇总2.1 Required任务 : 5/10 通过
2.2 可选任务 — 21/25 通过
3 失败详情（仅 required）Approval — 基础设施（审批门控）（置信度: 高）Approval
根因详情: 关键日志: 修复建议:
修复建议摘要: 联系RD成员(qingqing01/jiangjiajun/heavengate)进行审批 关联变更: PR 所有变更，目标分支 release/2.6 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/2.6 #7772 +/- ##
==============================================
Coverage ? 71.86%
==============================================
Files ? 378
Lines ? 53924
Branches ? 8432
==============================================
Hits ? 38750
Misses ? 12409
Partials ? 2765
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
3 participants
Motivation
Fix a bug in KSM when overlap is enabled. Since KSM itself introduces synchronization under overlap mode, which breaks the overlap, we temporarily do not consider asynchrony here and only ensure functional correctness.
Modifications
accept_num_cputoaccept_numto directly use the GPU-side accept_num for calculationUsage or Command
N/A
Accuracy Tests
N/A
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.