Add DRAM KV offloading (HiCache) to GLM-5.2 NVFP4 B300 SGLang agentic recipe#2279
Conversation
Run 29651235293 showed the 1M-context corpus working set outgrowing the HBM KV pool past conc 8 (TP8) / conc 64 (DP8): gpu_kv_cache_usage pinned at 1.0 while the radix hit rate collapsed from a ~0.97 theoretical ceiling to 0.04-0.06, so every post-knee turn re-prefilled its whole history and throughput fell together with interactivity, leaving the frontier a single dominated cliff. Spill evicted prefixes to a per-rank pinned host pool (0.80 x node DRAM / TP) instead of recomputing them, following the qwen3.5 b300 hicache recipe; GLM-5.2 is plain GQA so there is one host pool per rank and GB-based --hicache-size works (the DSv4 ratio-based sizing is a DSv4-only workaround). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — the HiCache DRAM-offload wiring is correct and consistent with the existing qwen3.5 hicache recipe pattern. The one reported issue is a minor process nit (missing bilingual PR title/description), not a functional bug.
Extended reasoning...
Overview
This PR removes the hard KV_OFFLOADING != none guard in glm5.2_fp4_b300_sglang.sh and wires up SGLang HiCache DRAM KV offloading for both search-space arms of the GLM-5.2 NVFP4 B300 agentic recipe, plus the corresponding configs/nvidia-master.yaml search-space update and an append-only perf-changelog.yaml entry.
Security risks
None. This is benchmark harness/config code with no auth, crypto, or externally-exposed surface — it only affects internal sweep runs.
Level of scrutiny
I verified the new require_agentic_kv_offload_backend hicache gate matches the existing helper in benchmarks/benchmark_lib.sh, and that the per-rank sizing math (TOTAL_CPU_DRAM_GB / TP) is correct given that TOTAL_CPU_DRAM_GB is already utilization-adjusted upstream in utils/matrix_logic/generate_sweep_configs.py::agentic_dram_offload_gb (applies dram-utilization and the gpu_count/gpus_per_node ratio before the script ever sees it) — so the script correctly just divides by TP for the per-rank pool, matching the PR description's math (2399/8 ≈ 299 GB/rank). The new flags mirror the referenced qwen3.5-fp8-b300-sglang-agentic-hicache recipe. This is a moderate-scrutiny benchmark-recipe change (not production-critical, no user-facing or security surface), and the logic checks out against the codebase as it stands today.
Other factors
The only finding from this run is a nit about the PR title/description missing the required bilingual Chinese section per AGENTS.md — a process/compliance gap, not a code defect. Given the underlying script and config changes are correct and follow an established pattern, this alone doesn't warrant blocking approval.
|
|
||
| - config-keys: | ||
| - glm5.2-fp4-b300-sglang-agentic | ||
| description: | ||
| - "Enable DRAM KV offloading (SGLang HiCache) on all points of both arms: per-rank pinned host pool = 0.80 x node DRAM / TP, page size 64, io-backend kernel, mem-layout page_first, write policy write_through_selective" | ||
| - "Run 29651235293 showed the 1M-context corpus working set outgrowing the HBM KV pool past conc 8 (TP8) / conc 64 (DP8): gpu_kv_cache_usage pinned at 1.0 and the radix hit rate collapsed from a ~0.97 theoretical ceiling to 0.04-0.06, so every post-knee turn re-prefilled its full history and throughput fell together with interactivity" |
There was a problem hiding this comment.
🟡 This PR's title and description are English-only, but AGENTS.md (line 7) mandates that every PR title follow the <English title> / <中文标题> format and that the body include a Chinese translation section (e.g. ## 中文说明). Please add the missing Chinese title suffix and translation section.
Extended reasoning...
AGENTS.md explicitly states: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary; ...). This applies to every PR and every issue..."
This PR's title is plain English: "Add DRAM KV offloading (HiCache) to GLM-5.2 NVFP4 B300 SGLang agentic recipe" — there is no / <中文标题> suffix. The description body contains only ## Summary and ## Validation sections, both in English, with no ## 中文说明 (or equivalent) section anywhere.
Why existing tooling doesn't catch this: the repo has a codeowner-signoff-verify.yml workflow that checks for the exact English sign-off phrase ("As a PR reviewer and CODEOWNER, I have reviewed this and have"), but that workflow verifies reviewer sign-off content, not author-side PR title/description bilingual formatting — there's no automated gate that rejects an English-only PR title or body before merge.
Step-by-step proof of the violation:
- Open AGENTS.md and read line 7 — it defines the exact required title format
<English title> / <中文标题>and requires a Chinese translation section in the body. - Read this PR's actual title: "Add DRAM KV offloading (HiCache) to GLM-5.2 NVFP4 B300 SGLang agentic recipe". Compare against the required format — there is no
/ <中文标题>suffix. - Read this PR's description — it has
## Summaryand## Validation, both English, no Chinese section. - Conclude: the required bilingual elements (title suffix + body translation section) are both absent, which is a direct, checkable violation of an explicit repo-mandated rule (not a matter of style/subjectivity).
Impact: Because this is a process/documentation compliance rule rather than something that affects the benchmark scripts, config, or changelog data added in this PR (benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh, configs/nvidia-master.yaml, perf-changelog.yaml all look functionally correct), it does not cause any runtime failure, incorrect benchmark behavior, or crash. It is a compliance gap against an explicit repo convention.
Fix: rename the PR title to append / <中文标题> with an idiomatic Chinese translation, and add a ## 中文说明 section to the description mirroring the ## Summary (and optionally ## Validation) content, following the translation quality bar described later in AGENTS.md (idiomatic technical Chinese, preserving model/hardware/framework names and flags in English).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29678598595 |
Run 29678598595 OOM-killed both arms at scheduler init: GLM-5.2 runs attention_backend=dsa (DeepSeek-sparse family; the 169.98 GB device KV pool is replicated on every rank), so the GB-based --hicache-size of TOTAL_CPU_DRAM_GB/TP pinned the whole 0.80-DRAM budget (8 x 299 GB = 2.4 TB) on top of 465 GB of weights on a 3 TB node. Control host capacity through the host/device token-capacity ratio like the DSv4 recipe — but DSv4's ratio=2 default also OOMs here (2 x 170 GB x 8 = 2.7 TB; GLM-5.2's device pool is far larger than DSv4's), so use fractional ratio 0.75 = ~128 GB/rank = ~1.0 TB total, matching the cluster's proven host-pool envelope. Adopt DSv4's write_back / direct / page_first_direct transfer flags. Validated on-node (b300-019, 2026-07-19): both sizing regressions reproduced, ratio 0.75 boots in ~290 s and survives a 4.2M-token overflow bench that forces eviction through the DSA KV+INDEXER host pools. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29682242847 |
1 participant
Summary
Follow-up to #2268. Run 29651235293 showed both arms falling off a cliff once the 1M-context agentic corpus working set outgrows the HBM KV pool:
gpu_kv_cache_usage_pctpins at 1.0 past conc 8 (TP8 arm) / conc 64 (DP8 attention-DP arm), and the radix hit rate collapses from a ~0.97 theoretical ceiling to 0.04–0.06. Every post-knee turn then re-prefills its entire 100k+-token history (at conc 32: 2,350 input tok/s/gpu vs 14.5 output tok/s/gpu), so throughput drops instead of plateauing while TTFT blows out to minutes — all post-knee points land dominated in the low-interactivity/low-throughput corner.This enables DRAM KV offloading on all points of both arms so evicted prefixes spill to a pinned host pool and come back at C2C bandwidth instead of being recomputed:
kv-offloading: dram+kv-offload-backend: hicacheon both search-space arms (conc lists unchanged)qwen3.5-fp8-b300-sglang-agentic-hicacherecipe:--page-size 64,--hicache-io-backend kernel,--hicache-mem-layout page_first, write policywrite_through_selective(overridable)TOTAL_CPU_DRAM_GB / TP(= 2399/8 ≈ 299 GB/rank at the configured 0.80 dram-utilization); GLM-5.2 is plain GQA so there is one host pool per rank and GB-based--hicache-sizeworks (the DSv4 ratio-based sizing is a DSv4-only workaround)pr-linkfilled in the follow-up commitValidation
process_changelog.py --base-ref origin/main --head-ref HEADemits all 13 benchmark rows + the SWE-benchagentic_evalsrow withkv-offloading: dram, backendhicache,total-cpu-dram-gb: 2399pytest utils/matrix_logic/test_validation.py utils/test_process_changelog.py: 135 passedbash -non the recipe scriptsweep-agentic-evalsjob confirmed to plumbKV_OFFLOAD_BACKEND(run-sweep.yml:670)🤖 Generated with Claude Code