[Cherry-Pick][Feature] support decode attention for mix(#7688)#7729
[Cherry-Pick][Feature] support decode attention for mix(#7688)#7729lizhenyun01 wants to merge 14 commits into
Conversation
|
Thanks for your contribution! |
lizhenyun01
changed the title
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览
2 任务状态汇总2.1 Required任务 : 3/8 通过
2.2 可选任务 — 16/20 通过
3 失败详情（仅 required）Approval — 代码规范（审批流程）（置信度: 高）Approval
根因详情: 关键日志: 修复建议:
修复建议摘要: 请指定RD对PR进行Review并批准（FastDeploy RD + PaddlePaddle RD） 链接: 查看日志 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release/2.6 #7729 +/- ##
==============================================
Coverage ? 72.45%
==============================================
Files ? 381
Lines ? 54016
Branches ? 8445
==============================================
Hits ? 39139
Misses ? 12084
Partials ? 2793
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-11 21:30:39
📋 Review 摘要
PR 概述：为 decode 阶段新增 C16（fp16/bf16 KV cache）和静态 C8（int8 KV cache）decode attention CUDA kernel，通过环境变量 USE_DECODE_ATTENTION=1 启用，仅支持 SM ≥ 90（Hopper+）+ NVCC ≥ 12.0。
变更范围：custom_ops/gpu_ops/append_attention/、layers/attention/、worker/gpu_model_runner.py、spec_decode/mtp.py
影响面 Tag：[OP] [Feature]
📝 PR 规范检查
标题格式符合 [Cherry-Pick][Feature] 描述(#原PR号) 规范 ✅。但 ## Modifications、## Usage or Command、## Accuracy Tests 三个 section 均为空（仅保留注释占位符），Checklist 条目未勾选，建议补全。
标题建议（可直接复制）：
[Cherry-Pick][Feature] support decode attention for C16/C8 mix(#7688)
PR 描述建议（可直接复制）：
## Motivation
C16/静态C8 attention支持，使用方式：flash_attn开启情况下export USE_DECODE_ATTENTION=1
## Modifications
- 新增 C16（fp16/bf16 KV cache）和静态 C8（int8 KV cache）decode attention CUDA 实现（`custom_ops/gpu_ops/append_attention/` 目录，含 attention_func.cuh、decode_append_attention_c16_impl.cuh、decode_append_attention_c8_impl.cuh 等核心 kernel 文件）
- 新增 `decode_append_attention.cu`、`decoder_write_cache_with_rope.cu` 作为算子入口，仅在 SM ≥ 90 且 NVCC ≥ 12.0 时编译
- 新增 Python wrapper 层：`layers/attention/ops/decode_append_attention.py`、`decoder_write_cache_with_rope.py`、`config_for_attention.py`
- 在 `flash_attn_backend.py` decode 阶段，当 `USE_DECODE_ATTENTION=1` 时切换到新 decode attention 路径（替代原 `append_attention`）
- `fastdeploy/envs.py` 新增 `USE_DECODE_ATTENTION` 环境变量开关
- 同步更新 `spec_decode/mtp.py`、`worker/gpu_model_runner.py`、`worker/metax_model_runner.py` 中的 decode attention buffer 分配与初始化
- 新增测试：`tests/operators/attention/test_decode_append_attention.py`、`test_decode_append_attention_c16.py`
## Usage or Command
```bash
export USE_DECODE_ATTENTION=1
# 在 flash_attn 开启时生效，仅限 SM ≥ 90（Hopper+），NVCC ≥ 12.0
```
## Accuracy Tests
N/A（该 PR 通过新 decode attention kernel 替代原 append_attention decode 路径，需要补充精度对比数据或说明无需精度验证的理由）
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [x] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [x] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 📝 PR 规范 | — | Modifications / Usage / Accuracy Tests 为空，Checklist 未勾选 |
| 🟡 建议 | fastdeploy/model_executor/layers/attention/flash_attn_backend.py:281 |
遗留调试 print 语句，应移除或改用 logger |
| 🟡 建议 | custom_ops/setup_ops.py:547 |
os.system() 返回值被忽略，代码生成失败时构建不中断 |
| ❓ 疑问 | fastdeploy/envs.py:281 |
ENABLE_V1_KVCACHE_MANAGER 在本次变更文件中未见引用，请确认是否遗漏或提前注册 |
总体评价
整体实现思路完整，新增 CUDA kernel + Python wrapper + 测试的闭环结构良好，SM 门控和环境变量开关设计合理。主要问题是遗留的调试 print、构建脚本错误不中断、以及 ENABLE_V1_KVCACHE_MANAGER 的用途需要确认；PR 描述需补充变更说明和精度验证数据。
| self.max_tokens_per_batch: int = self.speculate_max_draft_token_num + 1 | ||
| if FLASH_ATTN_VERSION is None: | ||
| init_flash_attn_version() | ||
| print(f"num_heads: {self.num_heads}, kv_num_heads: {self.kv_num_heads}") |
There was a problem hiding this comment.
🟡 建议 遗留调试 print 语句，会在每次 FlashAttnBackend 初始化时打印到 stdout，建议移除或改为 logger.debug()。
# 移除此行，或改为：
logger.debug("num_heads: %d, kv_num_heads: %d", self.num_heads, self.kv_num_heads)|
|
||
| if cc >= 90 and nvcc_version >= 12.0: | ||
| # decode attention | ||
| os.system( |
There was a problem hiding this comment.
🟡 建议 os.system() 返回值被丢弃。若 auto_gen_template_attention.py 执行失败（依赖缺失、模板错误等），后续 find_end_files("gpu_ops/append_attention", ".cu") 可能找不到自动生成的 .cu 文件，导致编译静默失败。建议检查退出码：
ret = os.system(
"python utils/auto_gen_template_attention.py --config ..."
)
if ret != 0:
raise RuntimeError("auto_gen_template_attention.py failed with code %d" % ret)| # Whether to enable FP8 quantization with pow2scale. | ||
| "FD_FP8_QUANT_WITH_POW2SCALE": lambda: bool(int(os.getenv("FD_FP8_QUANT_WITH_POW2SCALE", "0"))), | ||
| # enable kv cache manager v1 | ||
| "ENABLE_V1_KVCACHE_MANAGER": lambda: int(os.getenv("ENABLE_V1_KVCACHE_MANAGER", "0")), |
There was a problem hiding this comment.
❓ 疑问 ENABLE_V1_KVCACHE_MANAGER 在本次 PR 改动的文件中未见任何引用。请确认：
- 该变量是否确实属于本次变更范围，还是误混入？
- 若有使用点，是否遗漏了对应文件的变更？
3 participants
Motivation
C16/静态C8 attention支持，使用方式：flash_attn开启情况下export USE_DECODE_ATTENTION=1
Modifications
Usage or Command
Accuracy Tests
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.