[AMD][MI355X] DSv4 SGLang agentic: v0.5.14 image, router, HiCache by seungrokj · Pull Request #2146 · SemiAnalysisAI/InferenceX

seungrokj · 2026-07-10T06:51:48Z

Summary

Bump dsv4-fp4-mi355x-sglang-agentic image to lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
Align launcher env vars and server args with fixed-seq-len sibling (dsv4 attention backend, fp8_e4m3 KV cache, --disable-radix-cache, --cuda-graph-max-bs, DP-attention exports, --enable-two-batch-overlap)
Add SGLang router for DP-attention configs (consistent-hashing, dp-aware, correlation-id routing)
Add HiCache KV offloading support
Add SGLANG_CMD array pattern with command logging and env-var dump
Add capture_cache_metrics for pre/post-benchmark cache stats
Sweep conc=48 across TP8 +/- DPA +/- HiCache

Test plan

Trigger dsv4-fp4-mi355x-sglang-agentic full sweep on MI355X cluster
Verify TP8 (no DPA, no HiCache) completes without errors
Verify TP8+DPA (router) completes without errors
Verify TP8+HiCache completes without errors
Verify TP8+DPA+HiCache completes without errors

🤖 Generated with Claude Code

…th router and HiCache support Align the agentic launcher with the fixed-seq-len sibling: dsv4 attention backend, fp8_e4m3 KV cache, disable-radix-cache, DP-attention env exports (SGLANG_SHARED_EXPERT_TP1, DP_USE_GATHERV, etc.), and two-batch-overlap. Add SGLang router for DP-attention configs and HiCache KV offloading. Sweep conc=48 across TP8 +/- DPA +/- HiCache. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-10T06:51:56Z

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

claude · 2026-07-10T06:59:56Z

+  *)
+    echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: native, mooncake, lmcache)" >&2
+    exit 1


🟡 The default arm of the KV_OFFLOAD_BACKEND case at line 85 prints (expected: native, mooncake, lmcache), but the case above only accepts hicache — the listed values look copy-pasted from the sibling vLLM launchers (which do accept native/mooncake/lmcache) and would themselves fall into the same default arm here. Sweep configs only use hicache so CI never trips this branch, but anyone debugging by hand will be misdirected. Fix: change the list to (expected: hicache).

Extended reasoning...

What the bug is

The case "${KV_OFFLOAD_BACKEND:-}" block at lines 56-88 of benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh has exactly one accepting arm: hicache). Anything else falls into the default *) arm at lines 84-87, which emits:

Error: unsupported KV_OFFLOAD_BACKEND '<value>' (expected: native, mooncake, lmcache)

None of native, mooncake, or lmcache are recognized by this case statement — they are all backends supported by other launchers in the repo (native for kimik2.5-fp4-mi355x-vllm-agentic, mooncake for minimaxm3-fp8-mi300x-vllm-agentic, lmcache for the sibling dsv4-fp4-mi355x-vllm-agentic). The message is a copy-paste from those vLLM-family launchers.

Why the message is self-contradictory

A user who sees unsupported KV_OFFLOAD_BACKEND 'foo' and reads the parenthetical hint will pick one of native, mooncake, or lmcache — and hit the exact same default arm and the exact same message. In the case where they picked native, the output literally reads unsupported KV_OFFLOAD_BACKEND 'native' (expected: native, mooncake, lmcache), which is self-contradictory.

Step-by-step proof

Operator invokes the launcher manually with KV_OFFLOADING=dram KV_OFFLOAD_BACKEND=native (a natural attempt after seeing another dsv4 launcher use lmcache).

agentic_kv_offload_enabled returns true because KV_OFFLOADING=dram.

Bash enters the case at line 56 and evaluates the arms in order. The only arm is hicache).

native does not match hicache), so control falls to *) at line 84.

Line 85 prints: Error: unsupported KV_OFFLOAD_BACKEND 'native' (expected: native, mooncake, lmcache).

Line 86 exits 1.

Operator, taking the message at face value, retries with mooncake or lmcache. Same path — steps 3-6 repeat.

Why existing code doesn't prevent it

The header comment at line 7 states "KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=hicache", so the canonical documented value is hicache. The error text just wasn't updated to match when this launcher was adapted from the vLLM sibling.

Impact

Cosmetic misdirection in a diagnostic-only path. The four sweep entries in configs/amd-master.yaml for this recipe all pass kv-offload-backend: hicache, so CI never hits the default arm. The cost is a minute or two of wasted debug time for someone experimenting with a non-hicache backend by hand.

Fix

Change line 85's parenthetical to (expected: hicache):

echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: hicache)" >&2

On the refutations

Both refutations are duplicate-management: bug_002 and bug_004 target the same defect at the same lines with the same fix, and the refuters explicitly note the finding itself is real — they only refuted one of the two to avoid posting the same comment twice. The synthesis agent merged them into a single report, which resolves that concern.

…config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-10T07:35:57Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29076727890
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29076727890

…figs for initial validation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-10T08:48:39Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29077198891
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29077198891

…unning-requests, re-enable all sweep configs - Fix attention-backend typo (compresseed -> compressed) - Remove duplicate CUDA_GRAPH_MAX_BS block - Set MAX_RUNNING_REQUESTS to 2*CONC for subagent fan-out headroom - Use MEM_FRACTION_STATIC=0.90 and swa-full-tokens-ratio=0.10 - Add METRICS_ARGS array, guard capture_cache_metrics on metrics enabled - Remove duplicate SGLANG_ENABLE_UNIFIED_RADIX_TREE export from hicache block - Uncomment all 4 sweep configs (TP8 +/- DPA +/- HiCache) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-10T08:56:36Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081203249
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29081203249

github-actions · 2026-07-10T11:27:01Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081361033
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29081361033

…ust sweep concurrency hicache-ratio=8 tried to pin ~495 GB via cudaHostRegister, exceeding ROCm's per-process pinnable memory limit. Lower to ratio=4 and sweep graduated concurrency levels across configs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-10T16:32:00Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29094196523
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29094196523

…concurrency to 96 Device KV utilization peaked at only 27% even at conc=48, and non-DPA configs underperformed DPA in both throughput and completed requests. Drop non-DPA sweep entries and extend DPA concurrency range to [16..96] (none) and [48..96] (hicache) to find the saturation point. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-10T16:50:16Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29108708524
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29108708524

functionstackx

please update sglang cookbook with ur current optimized recipes, sglang cookbook has support for HiCache

github-actions · 2026-07-11T00:06:04Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29108764431
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29108764431

… conc 1-8) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-11T02:07:36Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29108764431
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29108764431

github-actions · 2026-07-11T02:17:49Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136063462
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29136063462

github-actions · 2026-07-11T02:21:31Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136075686
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29136075686

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-11T02:28:44Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136311108
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29136311108

…c hicache recipe Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-11T10:10:14Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136685794
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29136685794

github-actions · 2026-07-11T12:10:45Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29136685794
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29136685794

seungrokj · 2026-07-11T13:00:36Z

/reuse-sweep-run

seungrokj · 2026-07-11T13:00:58Z

@functionstackx can you plz review this?

functionstackx · 2026-07-13T01:23:38Z

@functionstackx can you plz review this?

would u like to address my last comment? #2146 (review)

seungrokj · 2026-07-13T09:00:05Z

@functionstackx here it is
sgl-project/sglang#31010
will ask Hai to merge this

functionstackx · 2026-07-13T23:49:28Z

@functionstackx here it is sgl-project/sglang#31010 will ask Hai to merge this

@seungrokj this seems like an issue and not an PR?

cquil11 · 2026-07-13T23:55:20Z

+        DEFAULT_HICACHE_RATIO=4
+
+        HICACHE_RATIO="${HICACHE_RATIO:-$DEFAULT_HICACHE_RATIO}"
+        HICACHE_WRITE_POLICY="${HICACHE_WRITE_POLICY:-write_through}"


instead of this bash syntax where you set default, just hard code. this file is the source of truth and nothing will ever override these env vars

this syntax ugly and slightly confusing

cquil11 · 2026-07-13T23:58:50Z

+case "${KV_OFFLOAD_BACKEND:-}" in
+  hicache)
+        # ---- Hicache config ----------------------------------------------------------
+        # DSv4 allocates several physical host sub-pools for each logical host


can we try to clean up the ai comments a bit. the comment may be necessary but just make it more succinct and make sure it is still needed as a human reading it. if necessary in current form, that fine

cquil11 · 2026-07-13T23:59:06Z

+        # both of which blow the limit. Ratio=2 keeps pinned memory safely
+        # under the ceiling while still providing a meaningful CPU KV tier.
+
+        # RuntimeError: cudaHostRegister failed (rc=1, invalid argument) for ptr=0x6f6388400000 size=531470745600; host buffer is not pinned and device transfers may silently return stale data 


like for insqtance is this needed or was this just an error that is now fixed?

seungrokj requested a review from a team July 10, 2026 06:51

seungrokj requested review from billishyahao, chunfangamd and yctseng0211 as code owners July 10, 2026 06:51

github-project-automation Bot added this to InferenceMAX Board Jul 10, 2026

seungrokj requested a review from 1am9trash as a code owner July 10, 2026 06:51

seungrokj added AMD full-sweep-fail-fast agentx AgentX benchmarks, recipes, and infrastructure labels Jul 10, 2026

claude Bot reviewed Jul 10, 2026

View reviewed changes

seungrokj and others added 2 commits July 10, 2026 16:24

[AMD][MI355X] Add dram-utilization 0.80 to DSv4 SGLang agentic sweep …

3fe2c5e

…config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Update perf-changelog with PR link for DSv4 SGLang agentic entry

e282bf7

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

[AMD][MI355X] Remove --context-length and comment out extra sweep con…

6acad04

…figs for initial validation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

seungrokj added full-sweep-enabled and removed full-sweep-fail-fast labels Jul 10, 2026

seungrokj and others added 2 commits July 11, 2026 01:48

Merge branch 'main' into amd/agentx_dsv4_sgl

f504c26

functionstackx requested changes Jul 10, 2026

View reviewed changes

[AMD][MI355X] DSv4 SGLang agentic: add non-DPA baseline config (tp=8,…

2294501

… conc 1-8) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge branch 'main' into amd/agentx_dsv4_sgl

d3c419f

seungrokj added 2 commits July 11, 2026 11:16

Update perf-changelog.yaml

0c6bcc3

Merge branch 'main' into amd/agentx_dsv4_sgl

9fa5ad9

[AMD][MI355X] exclude mia1-p01-g12 from agentic salloc

a91aa69

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

[AMD][MI355X] DSv4 SGLang: narrow DPA sweep ranges, add disagg agenti…

a1da9ec

…c hicache recipe Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge branch 'main' into amd/agentx_dsv4_sgl

173ee97

seungrokj requested a review from functionstackx July 13, 2026 00:19

seungrokj mentioned this pull request Jul 13, 2026

[Playground] Verified cell: mi355x / pro / fp4 / high-throughput / single sgl-project/sglang#31010

Open

3 tasks

seungrokj added 4 commits July 14, 2026 07:57

Merge branch 'main' into amd/agentx_dsv4_sgl

d252a2e

Update amd-master.yaml

7fcbd7c

Merge branch 'main' into amd/agentx_dsv4_sgl

e94f0d1

Update amd-master.yaml

479ed31

chore(config): add serving component metadata

6541d71

cquil11 reviewed Jul 13, 2026

View reviewed changes

Uh oh!

Conversation

seungrokj commented Jul 10, 2026

Summary

Test plan

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

Uh oh!

claude Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

functionstackx left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

seungrokj commented Jul 11, 2026

Uh oh!

seungrokj commented Jul 11, 2026

Uh oh!

functionstackx commented Jul 13, 2026

Uh oh!

seungrokj commented Jul 13, 2026

Uh oh!

functionstackx commented Jul 13, 2026

Uh oh!

cquil11 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

cquil11 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

cquil11 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

cquil11 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants