feat(runtime): execute host environment recipes#1671
Merged
Conversation
Deploying agentv with
|
| Latest commit: |
a83057b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e6532dbd.agentv.pages.dev |
| Branch Preview URL: | https://environment-host-runtime.agentv.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Host environment recipes are now executable at eval runtime. A
type: hostenvironment creates its resolvedworkdir, runs optional setup from the recipe source directory with typed JSON args on stdin, and makes that workdir the cwd for target providers and supported script graders.This keeps testbed setup on the eval environment instead of the selected target. Generic CLI targets and the Pi CLI coding-agent path now receive the prepared host workdir through the existing provider cwd contract, while Docker recipes remain schema-only with a clear unsupported-runtime error.
Related: av-noh3.2.3
Validation
bun test packages/core/test/evaluation/environment-host-runtime.test.ts packages/core/test/evaluation/loaders/environment-recipe.test.ts packages/core/test/evaluation/providers/cli.test.ts packages/core/test/evaluation/providers/pi-runtime.test.tsbun test ./packages/core/test/evaluation/graders.test.ts --grep "passes workspace_path"bun test ./apps/cli/test/commands/eval/bundle.test.ts --grep "bundles inherited"bun --filter @agentv/core testbun --filter agentv test -- --grep "agentv eval bundle"bunx biome check apps/cli/src/commands/eval/task-bundle.ts apps/cli/test/commands/eval/bundle.test.ts packages/core/src/evaluation/environment/host.ts packages/core/src/evaluation/graders/script-grader.ts packages/core/src/evaluation/loaders/environment-recipe.ts packages/core/src/evaluation/workspace/setup.ts packages/core/test/evaluation/environment-host-runtime.test.ts packages/core/test/evaluation/graders.test.ts packages/core/test/evaluation/loaders/environment-recipe.test.ts packages/core/test/evaluation/providers/pi-runtime.test.tsbun --filter @agentv/core typecheckbun --filter @agentv/core buildpi-cli: run2026-07-05T09-42-10-563Zpassed 1/1 with score 100%, target cwd set to the host environment workdir, and live grader acceptedHOST_ENV_READY.28736988713: Build, Check Links, Cloudflare Pages, Lint, Test, Typecheck, Validate Evals, and Validate Marketplace all passed.Evidence
Private evidence branch:
EntityProcess/agentv-private@997584c9066fd1dabaf69f9160c3931c78f01584onevidence/av-noh3-2-3-host-runtime.Review Notes
ce-code-reviewbecause generic subagent spawning is gated to explicit delegation requests in this harness. The scan found and fixed shared environment/workspace owner-key interaction before this PR was opened.Post-Deploy Monitoring & Validation
Watch early CI and any downstream dogfood runs that use
environment.type: hostwith setup commands. Healthy signals are successful setup execution, providertarget_execution.command.cwdmatching the resolvedenvironment.workdir, bundled eval YAML omitting internalsource_dirmetadata, and script grader raw requests showing the same cwd when no explicit grader cwd is set. Failure signals areunsupported_environmentfor host recipes,environment_setup_erroron valid setup scripts, target cwd falling back to the eval repo root, or bundles containing absolute source paths. Rollback is reverting this PR; owner is the AgentV runtime worker for the first 24 hours after merge.