fix: recover PR 2279 ingest via sweep reuse#2280
Conversation
Run 29651235293 showed the 1M-context corpus working set outgrowing the HBM KV pool past conc 8 (TP8) / conc 64 (DP8): gpu_kv_cache_usage pinned at 1.0 while the radix hit rate collapsed from a ~0.97 theoretical ceiling to 0.04-0.06, so every post-knee turn re-prefilled its whole history and throughput fell together with interactivity, leaving the frontier a single dominated cliff. Spill evicted prefixes to a per-rank pinned host pool (0.80 x node DRAM / TP) instead of recomputing them, following the qwen3.5 b300 hicache recipe; GLM-5.2 is plain GQA so there is one host pool per rank and GB-based --hicache-size works (the DSv4 ratio-based sizing is a DSv4-only workaround). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 29678598595 OOM-killed both arms at scheduler init: GLM-5.2 runs attention_backend=dsa (DeepSeek-sparse family; the 169.98 GB device KV pool is replicated on every rank), so the GB-based --hicache-size of TOTAL_CPU_DRAM_GB/TP pinned the whole 0.80-DRAM budget (8 x 299 GB = 2.4 TB) on top of 465 GB of weights on a 3 TB node. Control host capacity through the host/device token-capacity ratio like the DSv4 recipe — but DSv4's ratio=2 default also OOMs here (2 x 170 GB x 8 = 2.7 TB; GLM-5.2's device pool is far larger than DSv4's), so use fractional ratio 0.75 = ~128 GB/rank = ~1.0 TB total, matching the cluster's proven host-pool envelope. Adopt DSv4's write_back / direct / page_first_direct transfer flags. Validated on-node (b300-019, 2026-07-19): both sizing regressions reproduced, ratio 0.75 boots in ~290 s and survives a 4.2M-token overflow bench that forces eviction through the DSA KV+INDEXER host pools. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
/reuse-sweep-run 29682242847 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — this is an ingest-recovery PR with no code diff, just reusing artifacts from an already-completed sweep run (29682242847) since the push-to-main sweep on the merge commit ran without reuse authorized.
Extended reasoning...
Overview
This PR has no code diff. It exists solely to trigger ingest recovery for PR #2279's merge: the push-to-main sweep on the merge commit (run 29691795469) executed without /reuse-sweep-run authorized, so no official benchmark ingest landed for main. The /reuse-sweep-run 29682242847 comment reuses artifacts from the source pull_request sweep run on PR #2279 itself to complete ingestion, rather than re-running GPU benchmarks.
Security risks
None. This is a CI/data-ingest administrative action reusing artifacts already produced by a prior, already-validated sweep run. No new code, config, or logic is introduced.
Level of scrutiny
Low. This follows the established, repeated "recover-failed-ingest" pattern used elsewhere in this repo when a main-branch sweep fails to ingest — it is a mechanical recovery step, not a design or logic change.
Other factors
No bug reports were found (there is nothing to find, since there is no diff). No modified files were preloaded, consistent with this being a metadata/workflow-triggered ingest rather than a content change.
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Ingest recovery complete ✅Recovered the official ingest for PR #2279 ( Root cause (target that failed to ingest)
Source run reused
Recovery path
Recovered data
Verification
|
1 participant
Recover the missing official ingest from source run 29682242847 (pull_request run-sweep on PR #2279). The push-to-main sweep on the merge commit (run 29691795469) ran without
/reuse-sweep-runauthorized, so no official ingest landed. This PR reuses the source artifacts to complete ingestion.