feat(eval): run platform tasksets via /rollouts/run_list so traces get task names#489
Draft
MittelmanDaniel wants to merge 3 commits into
Draft
feat(eval): run platform tasksets via /rollouts/run_list so traces get task names#489MittelmanDaniel wants to merge 3 commits into
MittelmanDaniel wants to merge 3 commits into
Conversation
When a platform taskset is run wholesale on HostedRuntime with a stock
gateway model, submit one batch request (/rollouts/run_list) instead of
one /rollouts/submit per rollout. The server resolves the taskset's task
versions itself, so every trace is born linked to its named task and the
dashboard groups and labels them ("my-task (2)") instead of falling back
to trace UUIDs.
Anything the batch shape cannot express falls through to the existing
per-rollout path: an open job, a task subset (which drops api_id), a
custom model_client, agent config diverging from the type's defaults, or
a model absent from the platform catalog.
Fixes hud-evals#488
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four-lens review (conventions, correctness, concision, abstractions): - Cancel submitted traces on batch timeout, mirroring _await_terminal (previously leaked platform leases on TimeoutError). - Take the caller's tasks x group count as run_taskset(expected=...) so a lazily-populated submission response can't end the poll early; fold exactly the submitted traces (dedup strays) in submission order, and set run.group_id from each trace's task_version_id so group structure survives the batch path. Document the ordering deviation in Taskset.run's docstring. - Treat a trace with a stamped end_time as terminal even under an unrecognized status, so one odd status can't stall the whole batch. - Rename gateway_batch_spec -> _gateway_batch_spec (matches the module's private-helper convention, e.g. _declared_env/_declared_names). - Move catalog matching next to the catalog: resolve_gateway_model() in hud/utils/gateway.py; extract the platform list-envelope parsing into hud.utils.platform.list_items() and reuse it in cli/jobs.py. - Flatten the dispatch guard with a walrus; drop a dead 'or ""' branch; ruff format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ob.runs docs pyright flagged the untyped getattr result in _gateway_batch_spec, and the Job.runs reference table promised expansion order unconditionally — the hosted batch path can deviate, so say so where the contract is documented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #488
When a platform taskset is run wholesale on
HostedRuntimewith a stock gateway model, submit the batch as onePOST /rollouts/run_listinstead of one/rollouts/submitper rollout. The server resolves the task versions itself, so traces come back named and grouped on the dashboard instead of as UUID cards (see the issue for screenshots of the two).Anything the batch shape can't express falls through to the existing per-rollout path unchanged: an open
job, a task subset (which dropsapi_id), a custommodel_client, agent config diverging from the type's defaults, or a model the catalog doesn't know.Implementation notes:
HostedRuntime.run_tasksetposts once, then polls/jobs/{id}/tracesuntil the caller's tasks×group count is terminal. On timeout or cancel it cancels the submitted traces platform-side. Runs are folded in submission order, deduped against strays, withrun.group_idset from each trace'stask_version_idso group structure survives. The deviation fromTaskset.run's expansion-order promise on this path is documented in its docstring and in theJob.runsreference table._gateway_batch_specdecides expressibility: stock gateway agent whose config matches its type's defaults, model resolved via the catalog (resolve_gateway_model, added tohud/utils/gateway.py). One judgment call:auto_respondis tolerated as a divergence — the CLI's--fullalways sets it, it only steers the client-side loop, and on this path the platform runs its own loop either way (same as website batch runs). If it should fall back instead, that's a one-line change.cli/jobs.pyintohud.utils.platform.list_itemsand reused it here.Assumptions worth a look from someone who can see the backend:
/jobs/{id}/tracesis read as a single unpaginated page;run_timeoutbounds the whole batch on this path (fine while traces run concurrently, but different from the per-rollout meaning); and the whole approach leans onrun_listaccepting API-key auth, so it's worth confirming that's a stable public surface.Testing: new tests in
test_hosted.py(payload shape, polling/folding, ordering/dedup/group_id, scheduler dispatch, fallback for custom agent config); full check suite passes (pytest -q: 682 passed;ruff format --check,ruff check,pyrightclean on the touched files). Live A/B on my own taskset: job8dc2bd52-5621-419b-b6e4-204afb264100(this branch) has every trace linked to its task version and renders named/grouped; jobc0e16e7192af40bbb5305ade930b4434(stock path, minutes earlier) hastask_version_id: nullon all traces.The backend change sketched in #488 (option a) would cover the fallback cases too — this is the client-side half.