Skip to content

feat(eval): run platform tasksets via /rollouts/run_list so traces get task names#489

Draft
MittelmanDaniel wants to merge 3 commits into
hud-evals:mainfrom
MittelmanDaniel:cli-batch-run-list
Draft

feat(eval): run platform tasksets via /rollouts/run_list so traces get task names#489
MittelmanDaniel wants to merge 3 commits into
hud-evals:mainfrom
MittelmanDaniel:cli-batch-run-list

Conversation

@MittelmanDaniel

@MittelmanDaniel MittelmanDaniel commented Jul 10, 2026

Copy link
Copy Markdown

Fixes #488

When a platform taskset is run wholesale on HostedRuntime with a stock gateway model, submit the batch as one POST /rollouts/run_list instead of one /rollouts/submit per rollout. The server resolves the task versions itself, so traces come back named and grouped on the dashboard instead of as UUID cards (see the issue for screenshots of the two).

Anything the batch shape can't express falls through to the existing per-rollout path unchanged: an open job, a task subset (which drops api_id), a custom model_client, agent config diverging from the type's defaults, or a model the catalog doesn't know.

Implementation notes:

  • HostedRuntime.run_taskset posts once, then polls /jobs/{id}/traces until the caller's tasks×group count is terminal. On timeout or cancel it cancels the submitted traces platform-side. Runs are folded in submission order, deduped against strays, with run.group_id set from each trace's task_version_id so group structure survives. The deviation from Taskset.run's expansion-order promise on this path is documented in its docstring and in the Job.runs reference table.
  • _gateway_batch_spec decides expressibility: stock gateway agent whose config matches its type's defaults, model resolved via the catalog (resolve_gateway_model, added to hud/utils/gateway.py). One judgment call: auto_respond is tolerated as a divergence — the CLI's --full always sets it, it only steers the client-side loop, and on this path the platform runs its own loop either way (same as website batch runs). If it should fall back instead, that's a one-line change.
  • Extracted the list-envelope parsing that existed twice in cli/jobs.py into hud.utils.platform.list_items and reused it here.

Assumptions worth a look from someone who can see the backend: /jobs/{id}/traces is read as a single unpaginated page; run_timeout bounds the whole batch on this path (fine while traces run concurrently, but different from the per-rollout meaning); and the whole approach leans on run_list accepting API-key auth, so it's worth confirming that's a stable public surface.

Testing: new tests in test_hosted.py (payload shape, polling/folding, ordering/dedup/group_id, scheduler dispatch, fallback for custom agent config); full check suite passes (pytest -q: 682 passed; ruff format --check, ruff check, pyright clean on the touched files). Live A/B on my own taskset: job 8dc2bd52-5621-419b-b6e4-204afb264100 (this branch) has every trace linked to its task version and renders named/grouped; job c0e16e7192af40bbb5305ade930b4434 (stock path, minutes earlier) has task_version_id: null on all traces.

The backend change sketched in #488 (option a) would cover the fallback cases too — this is the client-side half.

When a platform taskset is run wholesale on HostedRuntime with a stock
gateway model, submit one batch request (/rollouts/run_list) instead of
one /rollouts/submit per rollout. The server resolves the taskset's task
versions itself, so every trace is born linked to its named task and the
dashboard groups and labels them ("my-task (2)") instead of falling back
to trace UUIDs.

Anything the batch shape cannot express falls through to the existing
per-rollout path: an open job, a task subset (which drops api_id), a
custom model_client, agent config diverging from the type's defaults, or
a model absent from the platform catalog.

Fixes hud-evals#488

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four-lens review (conventions, correctness, concision, abstractions):

- Cancel submitted traces on batch timeout, mirroring _await_terminal
  (previously leaked platform leases on TimeoutError).
- Take the caller's tasks x group count as run_taskset(expected=...) so
  a lazily-populated submission response can't end the poll early; fold
  exactly the submitted traces (dedup strays) in submission order, and
  set run.group_id from each trace's task_version_id so group structure
  survives the batch path. Document the ordering deviation in
  Taskset.run's docstring.
- Treat a trace with a stamped end_time as terminal even under an
  unrecognized status, so one odd status can't stall the whole batch.
- Rename gateway_batch_spec -> _gateway_batch_spec (matches the module's
  private-helper convention, e.g. _declared_env/_declared_names).
- Move catalog matching next to the catalog: resolve_gateway_model() in
  hud/utils/gateway.py; extract the platform list-envelope parsing into
  hud.utils.platform.list_items() and reuse it in cli/jobs.py.
- Flatten the dispatch guard with a walrus; drop a dead 'or ""' branch;
  ruff format.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MittelmanDaniel MittelmanDaniel changed the title feat(eval): submit whole platform tasksets via /rollouts/run_list (named traces on the dashboard) feat(eval): run platform tasksets via /rollouts/run_list so traces get task names Jul 10, 2026
…ob.runs docs

pyright flagged the untyped getattr result in _gateway_batch_spec, and the
Job.runs reference table promised expansion order unconditionally — the
hosted batch path can deviate, so say so where the contract is documented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hosted runs submitted via CLI lose task names on the dashboard (rollouts/submit vs rollouts/run_list)

1 participant