GLM-5.2 NVFP4 B300 SGLang high-throughput arm to DEP8, cap conc at 64 #2281
Conversation
…at 64 / 为 glm5.2 b300 sglang 高吞吐分支启用 DEP(EP8)并将并发上限设为 64 Add ep: 8 to the TP8/DP8 attention-DP arm of glm5.2-fp4-b300-sglang-agentic so the MoE layers run expert-parallel across all 8 ranks (--ep-size 8) instead of TP-sharded (the arm previously left ep unset -> EP_SIZE=1). Attention stays data-parallel behind sglang-router consistent hashing. The recipe already passes --ep-size EP_SIZE unconditionally, so only the master-config search space changes; no recipe logic change is needed. Cap the DEP arm at conc 64 (conc-list [48, 64], was up to 512). The low-latency TP8 arm (conc 1-32, fp8 KV + cutedsl GEMM) is unchanged. DEP mirrors the proven dsv4-fp4-b300-sglang tp8/ep8/dp-attn arm. 中文：为 glm5.2-fp4-b300-sglang-agentic 的 TP8/DP8 注意力数据并行(attention-DP) 分支添加 ep: 8，使 混合专家(MoE) 层在全部 8 个 rank 上按 专家并行(EP) 运行 (--ep-size 8)，而非此前的 TP 切分(未设置 ep -> EP_SIZE=1)。注意力仍为数据 并行，并置于 sglang-router 一致性哈希之后以保持会话亲和性。该 recipe 脚本本就 无条件传入 --ep-size EP_SIZE，故仅需修改 master 配置的搜索空间，无需改动脚本 逻辑。将 DEP 分支并发上限设为 64(conc-list [48, 64]，原先最高 512)。低延迟 TP8 分支(并发 1-32，fp8 KV + cutedsl GEMM)保持不变。此 DEP 配置对齐已验证的 dsv4-fp4-b300-sglang tp8/ep8/dp-attn 分支。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
…5.2 b300 DEP 条目的 perf-changelog pr-link 中文：将 glm5.2-fp4-b300-sglang-agentic DEP 条目的 pr-link 从占位符更新为 #2281。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| - config-keys: | ||
| - glm5.2-fp4-b300-sglang-agentic | ||
| description: | ||
| - "Convert the high-throughput arm to DEP: add ep: 8 to the TP8/DP8 attention-DP arm so the MoE layers run expert-parallel across all 8 ranks (--ep-size 8) instead of TP-sharded (the arm previously left ep unset -> EP_SIZE=1). Attention stays data-parallel behind sglang-router consistent hashing; the recipe already passes --ep-size EP_SIZE unconditionally, so only the master-config search space changes" | ||
| - "Cap the DEP arm concurrency at 64: conc-list [48, 64] (was [48, 64, 96, 128, 192, 256, 512])" | ||
| - "Low-latency TP8 arm (conc [1, 2, 4, 8, 16, 32], fp8 KV + cutedsl bf16 GEMM) unchanged; DEP mirrors the proven dsv4-fp4-b300-sglang tp8/ep8/dp-attn DEP arm" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PENDING |
There was a problem hiding this comment.
🟡 The appended perf-changelog.yaml entry uses pr-link: .../pull/PENDING, which is not one of the two accepted placeholders (XXX or .../pull/XXX) in utils/validate_perf_changelog.py. This will cause prepare_perf_changelog_merge.py canonicalize (invoked by merge_with_reuse.sh) to raise ChangelogValidationError and block the merge until the placeholder is changed to XXX.
Extended reasoning...
The newly appended entry at the end of perf-changelog.yaml (lines 4969-4976) sets:
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PENDINGutils/validate_perf_changelog.py defines the only two accepted placeholder values while a PR is still open:
PR_LINK_PLACEHOLDERS = {
"XXX",
"https://github.com/SemiAnalysisAI/InferenceX/pull/XXX",
}and validate_added_pr_link() (around line 144-156) requires that a newly-added entry's pr-link be either one of those placeholders, or the canonical https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number} link matching the real PR number (CANONICAL_PR_LINK regex, /pull/\d+). .../pull/PENDING matches neither — it isn't in PR_LINK_PLACEHOLDERS, and PENDING doesn't satisfy \d+ — so this raises ChangelogValidationError.
Where this actually bites: not at PR-time CI. validate_perf_changelog.py::main() only runs validate_matrix_compatible_change → process_changelog.py, which never touches pr-link at all, and test-changelog-gate.yml only runs unit tests against fixtures. So the PR's own checks (run-sweep.yml's validate step, test-changelog-gate.yml) will pass as-is. The real failure surfaces at merge time: utils/merge_with_reuse.sh invokes prepare_perf_changelog_merge.py canonicalize --pr-number <N>, which calls canonicalize_appended_links() → compare_entries() → validate_added_pr_link(link, pr_number). With the real PR number (2281) supplied, pull/PENDING is neither a whitelisted placeholder nor equal to the expected canonical pull/2281 link, so validate_added_pr_link raises, and canonicalize_appended_links (lines ~112-115) also independently rejects it as "appended entry ... has unexpected pr-link". This hard-fails the standard merge_with_reuse.sh merge path for this PR.
Step-by-step proof:
- PR is opened as GLM-5.2 NVFP4 B300 SGLang high-throughput arm to DEP8, cap conc at 64 #2281; the appended changelog entry ships with
pr-link: .../pull/PENDING. - At merge time,
merge_with_reuse.shrunsprepare_perf_changelog_merge.py canonicalize --pr-number 2281 --base-ref <base>. - That calls
canonicalize_appended_links(..., pr_number=2281), which callscompare_entries(base, head, 2281). compare_entriesfinds the new entry and callsvalidate_added_pr_link("https://github.com/SemiAnalysisAI/InferenceX/pull/PENDING", 2281).- Inside
validate_added_pr_link:link not in PR_LINK_PLACEHOLDERS(true, since PENDING isn't XXX) andlink != expected(expected is.../pull/2281) → raisesChangelogValidationError. merge_with_reuse.shaborts, requiring a manual fix before the PR can be merged.
Fix: change pull/PENDING to XXX (the repo's documented placeholder convention per AGENTS.md and .github/workflows/claude.yml), which prepare_perf_changelog_merge.py will then correctly canonicalize to pull/2281 at merge time.
This does not fail the PR's own CI gate, so it doesn't block review/approval, but it will block the supported merge tooling until corrected — a small, mechanical fix.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29694367575 |
…recipe / 移除 glm5.2 b300 recipe 中为 conc 512 调优的 warmup grace 覆盖值 AGENTIC_WARMUP_GRACE_PERIOD=3600 was tuned for the conc-512 DP8 point, which this PR removes (DEP arm now caps at conc 64). At conc 64 the warmup working set drains well within the default 1800s (warmup-grace-period is a ceiling, not a fixed sleep), so fall back to the default and delete the now-stale comment. No behavioral change at conc <= 64. 中文：AGENTIC_WARMUP_GRACE_PERIOD=3600 是为 conc 512 的 DP8 数据点调优的，而本 PR 已移除该点(DEP 分支并发上限现为 64)。在 conc 64 下，warmup 工作集会在默认 1800s 内完成 drain(warmup-grace-period 是上限而非固定等待),故回退到默认值并 删除已过时的注释。conc <= 64 下无行为变化。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29717273540 |
…收敛为仅 conc 48 The sweep (run 29717273540) showed DEP conc 48 is the frontier peak (163 tok/s/gpu out, ttft 2.8s, GPU radix hit 0.93) while conc 64 overflows the HBM+host radix cache (hit collapses to 0.57), thrashing on re-prefill down to 41 tok/s/gpu at 56s ttft. conc 64 is strictly dominated by conc 48 on both throughput and interactivity and cannot be recovered without more host-pool capacity than the 3TB node can give (ratio > 0.75 OOMs, per #2279). Drop it: DEP arm conc-list is now [48]. The agentic eval also moves to conc 48 (off the conc-64 point whose eval hung). 中文：扫描(run 29717273540)显示 DEP conc 48 是前沿峰值(输出 163 tok/s/gpu， ttft 2.8s，GPU radix 命中 0.93)，而 conc 64 会使 HBM+host radix 缓存溢出(命中 率跌至 0.57)，陷入 re-prefill 抖动，降至 41 tok/s/gpu、ttft 56s。conc 64 在吞吐 与交互性两个维度上都被 conc 48 严格支配，且在 3TB 节点上无法通过增大 host pool 恢复(ratio > 0.75 会 OOM，见 #2279)。故移除：DEP 分支 conc-list 现为 [48]。 agentic 评估点也随之移到 conc 48(避开评估卡住的 conc 64 点)。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29741710665 |
|
/reuse-sweep-run |
Oseltamivir
changed the title
1 participant
Summary
Follow-up to #2268 / #2279. The GLM-5.2 NVFP4 B300 SGLang agentic recipe's high-throughput arm ran TP8 + DP8 attention-DP with the MoE layers TP-sharded — the arm left
epunset, soEP_SIZEdefaulted to1. This converts that arm to DEP (data-parallel attention + expert-parallel MoE) and runs it at conc 48 (the frontier peak).ep: 8added to the{ tp: 8, dp-attn: true, ... }throughput arm → the MoE layers now run expert-parallel across all 8 ranks (--ep-size 8) while attention stays data-parallel behind sglang-router consistent hashing. Mirrors the already-provendsv4-fp4-b300-sglangtp8/ep8/dp-attnDEP arm.conc-listis[48](was[48, 64, 96, 128, 192, 256, 512]). The validation sweep (run 29717273540) measured DEP conc-48 at 163 tok/s/gpu output / 2.8 s TTFT / 0.93 GPU radix hit — vs the previously-published dp-attn-ep1conc-48 at 124 / 5.3 s / 0.91, i.e. +31% output throughput from the EP change. Higher concurrencies are dropped (see the conc-64 note).conc [1, 2, 4, 8, 16, 32]with its fp8-KV / cutedsl bf16-GEMM levers stays exactly as-is (per the chosen scope — only the throughput arm becomes DEP).glm5.2_fp4_b300_sglang.shalready passes--ep-size "$EP_SIZE"unconditionally, so the DEP switch is master-config-only. The recipe is touched only to (a) update the header comment to describe the DEP arm and (b) drop the now-stale conc-512-tunedAGENTIC_WARMUP_GRACE_PERIOD=3600override — the DEP arm now runs conc 48, which drains within the default 1800 s grace (a ceiling, not a fixed sleep).The
EP_SIZE/DP_ATTENTIONplumbing already reaches the container:generate_sweep_configs.py(epdefaults to 1) →benchmark-tmpl.yml(EP_SIZE/DP_ATTENTIONjob env) →launch_b300-nv.sh(srun --export=ALL).Why conc 48 only (the conc-64 investigation)
conc 64 was dropped after investigating why it looked bad — the story had two distinct causes:
CUDA error: CUBLAS_STATUS_EXECUTION_FAILEDin the MoE router-gate GEMM (linear_bf16_fp32,deepseek_v2.pyMoEGate.forward), profiling only 129 records before AIPerf aborted on the error spike. This is the normal cuBLAS router-gate path for batches >max_router_gemm_tokens(16); it is not selected by--bf16-gemm-backend, so cutedsl would not have helped. It hit conc 48 and 64 on two different nodes while conc 96/128 ran the identical path clean — an intermittent cuBLAS/CUDA execution failure. The DEP rerun (29717273540) confirmed this was transient: conc 64 ran clean (1357 records, full duration, 0 crashes).So conc 64 is dominated on both axes and unrecoverable within the node's memory envelope → the DEP arm runs conc 48 only. The SWE-bench
agentic_evalspoint also moves to conc 48 (the conc-64 eval job hung ~4.5 h).Note / possible follow-up
Unlike the DSv4 B300 DEP arm (which adds
--moe-runner-backend flashinfer_mxfp4), this change does not add an explicit MoE-dispatch backend flag — SGLang's default single-node (NVLink) all-to-all dispatch is used.Validation
generate_sweep_configs.py full-sweep --model-prefix glm5.2→ 7 agentic points: 6 low-latency (ep=1, conc 1–32, unchanged) + 1 DEP (ep=8, dp-attn=true, router, conc 48); exp-names unique.process_changelog.py --base-ref origin/main --head-ref HEADemits the 7 rows + the SWE-benchagentic_evalsrow on the DEP conc-48 point,ep: 8on the DEP arm.pytest utils/matrix_logic/test_validation.py utils/test_process_changelog.py: 135 passed.bash -n benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh.中文说明
延续 #2268 / #2279。GLM-5.2 NVFP4 B300 SGLang agentic recipe 的高吞吐分支此前运行的是 TP8 + DP8 注意力数据并行(attention-DP)，而 混合专家(MoE) 层按 TP 切分——该分支未设置
ep，因此EP_SIZE默认为1。本次将该分支改为 DEP（注意力数据并行 + 专家并行(EP) MoE），并以 conc 48（前沿峰值）运行。ep: 8到{ tp: 8, dp-attn: true, ... }高吞吐分支 → MoE 层现按 专家并行 分布在全部 8 个 rank 上（--ep-size 8），注意力仍为数据并行并置于 sglang-router 一致性哈希之后。对齐已验证的dsv4-fp4-b300-sglangtp8/ep8/dp-attnDEP 分支。conc-list现为[48]（原先为[48, 64, 96, 128, 192, 256, 512]）。验证扫描(run 29717273540)测得 DEP conc-48 为 163 tok/s/gpu 输出 / 2.8s TTFT / 0.93 GPU radix 命中——对比此前发布的 dp-attn-ep1conc-48 的 124 / 5.3s / 0.91，即 EP 带来 +31% 输出吞吐。更高并发已移除（见 conc-64 说明）。conc [1, 2, 4, 8, 16, 32]，其 fp8-KV / cutedsl bf16-GEMM 配置完全保留（按所选范围，仅高吞吐分支改为 DEP）。glm5.2_fp4_b300_sglang.sh本就无条件传入--ep-size "$EP_SIZE"，故 DEP 切换仅涉及 master 配置。recipe 仅做两处修改：(a) 更新头注释以描述 DEP 分支；(b) 移除已过时、为 conc 512 调优的AGENTIC_WARMUP_GRACE_PERIOD=3600覆盖值——DEP 分支现运行 conc 48，可在默认 1800s grace 内完成 drain（grace 是上限而非固定等待）。EP_SIZE/DP_ATTENTION的传递链路已就绪：generate_sweep_configs.py（ep默认为 1）→benchmark-tmpl.yml（EP_SIZE/DP_ATTENTION任务级 env）→launch_b300-nv.sh（srun --export=ALL）。为何仅保留 conc 48（conc-64 调查）
在确认 conc 64 表现不佳的原因后将其移除，问题有两个独立成因：
linear_bf16_fp32，deepseek_v2.pyMoEGate.forward)抛出CUDA error: CUBLAS_STATUS_EXECUTION_FAILED，AIPerf 在错误率超阈后中止，仅 profile 了 129 条记录。这是批大小 >max_router_gemm_tokens(16) 时走的正常 cuBLAS 路由门路径，不受--bf16-gemm-backend控制，故 cutedsl 无法修复。它在两个不同节点上命中 conc 48 与 64，而 conc 96/128 走同一路径却正常完成——属于间歇性 cuBLAS/CUDA 执行失败。DEP 重跑(29717273540)证实其为偶发：conc 64 运行正常(1357 条记录，完整时长，0 崩溃)。因此 conc 64 在两个维度上均被支配，且在节点内存范围内无法恢复 → DEP 分支仅运行 conc 48。SWE-bench
agentic_evals点也随之移到 conc 48（conc-64 评估任务曾卡住约 4.5 小时）。备注 / 可能的后续
与 DSv4 B300 DEP 分支（额外添加
--moe-runner-backend flashinfer_mxfp4）不同，本次改动未显式添加 MoE 分发后端参数——使用 SGLang 默认的单节点(NVLink)全互联(all-to-all)分发。验证
generate_sweep_configs.py full-sweep --model-prefix glm5.2→ 7 个 agentic 点：6 个低延迟（ep=1，conc 1–32，未变）+ 1 个 DEP（ep=8, dp-attn=true, router，conc 48）；exp-name 唯一。process_changelog.py --base-ref origin/main --head-ref HEAD输出 7 行 + DEP conc-48 点上的 SWE-benchagentic_evals行，DEP 分支带ep: 8。pytest utils/matrix_logic/test_validation.py utils/test_process_changelog.py：135 passed。bash -n benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh。