[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups#2153
[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups#2153ApostaC wants to merge 15 commits into
Conversation
Signed-off-by: ApostaC <yihua98@uchicago.edu>
ApostaC
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
Signed-off-by: ApostaC <yihua98@uchicago.edu>
cquil11
added
the
agentx
ChatGPT Codex Connector
…ntry Resolve the perf-changelog.yaml tail conflict with #2127 per KLAUD_DEBUG.md §1.1: take main's changelog wholesale and re-append this PR's entry at the end, with pr-link updated from the XXX placeholder to #2153. 中文：将 origin/main 合并到 lmcache/agentix，按 KLAUD_DEBUG.md §1.1 处理 perf-changelog.yaml 末尾冲突：整体采用 main 的版本并在末尾重新追加本 PR 的 条目，同时把 pr-link 占位符更新为 #2153。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🟡
perf-changelog.yaml:4720-4727— PR title and description are English-only, but AGENTS.md requires bilingual PR metadata: title format<English title> / <中文标题>and a## 中文说明section in the body mirroring the English summary. Please update the title (e.g. append/ [LMCache] 为 dsv4 vllm b200/b300 agentix 配置添加 LMCache) and add a## 中文说明section to the description before merge.Extended reasoning...
AGENTS.md line 7 states: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: ' / <中文标题>'. In the PR/issue body, follow the English content with its Chinese translation (e.g. a '## 中文说明' section mirroring the summary)." This is a repo-wide convention that applies to every PR.
The PR under review is missing both required pieces:
- Title:
[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups— English only, no/ <中文标题>suffix. - Body: two English bullets under
# Summary("Adding LMCache configurations for dsv4-vllm launching script" / "Adding 2 nvidia configs with lmcache agentix") with no## 中文说明section following them.
Step-by-step proof of the violation:
- Open AGENTS.md at line 7 — the bilingual rule is explicit and unambiguous, with the exact title format and the
## 中文说明section pattern spelled out. - Read the PR title as rendered on GitHub — it contains only ASCII characters and no
/separator introducing a Chinese portion. - Read the PR body — the entire description is 3 lines of English (
# Summaryheader plus two bullets); grep for中文说明in the body returns nothing. - Therefore both required bilingual elements are absent, directly violating the rule at AGENTS.md:7.
Impact: This is metadata-only — the benchmark scripts and configs themselves are unaffected, CI will still run, and no runtime behavior changes. But the repo convention exists so that non-English speakers on the team can review and search PR history, and every merged PR sets precedent. The rule is stated as a requirement, not a suggestion.
How to fix (no code change needed):
- Edit the PR title to append a Chinese translation after
/, e.g.[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups / [LMCache] 为 dsv4 vllm b200/b300 agentix 配置添加 LMCache. - Add a
## 中文说明section to the PR body after the English summary, mirroring the two bullets — for example:## 中文说明 - 为 dsv4-vllm 启动脚本添加 LMCache 配置 - 新增 2 个使用 LMCache agentix 的 NVIDIA 配置
Severity is nit — this is a documentation/metadata policy issue, easy to fix in the PR description with no code impact, but should be addressed before merge since AGENTS.md declares it a requirement.
- Title:
…ntry Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29109297629 |
Bring-up on PR #2153 showed full-budget pinned pools OOM-killing B200 nodes (host OOM-killer / cudaErrorMemoryAllocation) as the LMCache lazy allocator filled past ~2 TB, so cap --l1-size-gb at 75% of TOTAL_CPU_DRAM_GB. Run the LMCache section on vllm/vllm-openai:v0.24.0 (the LMCache-0.5.x-validated pairing) with the CUTLASS DSL cu13 reinstall gated to vLLM < 0.24, and drop the B300 LMCache section for now to focus test runs on B200. 中文：PR #2153 启动调试发现满预算固定内存池会在缓存填充超过约 2 TB 时触发 节点 OOM（主机 OOM-killer / cudaErrorMemoryAllocation），故将 --l1-size-gb 上限降至 TOTAL_CPU_DRAM_GB 的 75%。LMCache 配置改用 vllm/vllm-openai:v0.24.0 镜像（与 LMCache 0.5.x 验证过的组合），CUTLASS DSL cu13 重装仅在 vLLM < 0.24 时执行；暂时移除 B300 的 LMCache 配置，将测试集中在 B200。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29109420492 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29116938835 |
| # The v0.24 images ship fixed bindings; reinstalling there would downgrade. | ||
| VLLM_INSTALLED_VERSION=$(python3 -c "from importlib.metadata import version; print(version('vllm'))") | ||
| if [ "$(printf '%s\n' "$VLLM_INSTALLED_VERSION" 0.24.0 | sort -V | head -n1)" != "0.24.0" ]; then | ||
| agentic_pip_install --quiet --force-reinstall --no-deps \ | ||
| 'nvidia-cutlass-dsl-libs-cu13==4.5.2' |
There was a problem hiding this comment.
generally we don't allow reinstalling kernel libraries
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
thanks for the contribution @ApostaC
can u please update recipes.vllm.ai with ur new configs? https://github.com/vllm-project/recipes
vllm recipes doesn't currently support offloading tho the inferact guys said they are open to contributions to support mooncake store, lmcache, vllm native offloading support in vllm recipes
as in sglang cookbook recipes , they also have support for cpu offloading recipes
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29117029396 |
| agentic_pip_install --quiet --no-cache-dir --no-deps \ | ||
| --force-reinstall "mooncake-transfer-engine-cuda13==$MOONCAKE_VERSION" | ||
| python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null | ||
| python3 "$(dirname "$0")/patch_vllm_pr45406.py" |
There was a problem hiding this comment.
generally we wanna avoid patches too
There was a problem hiding this comment.
Yeah, this is what I picked from #2138. Once that is in, I will remove this.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29128619300 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29133742844 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29133742844 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29164165383 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29168226910 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29170738763 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29174945007 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29177997711 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
| # AgentX concurrency counts live session trees, not individual requests. | ||
| # Subagent fan-out can push instantaneous request concurrency above CONC, so | ||
| # leave 2x headroom rather than clipping those bursts at the scheduler. | ||
| MAX_NUM_SEQS=$((2 * CONC)) | ||
|
|
There was a problem hiding this comment.
This is being moved to the top of the script
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29272097721 |
3 participants
Summary