diff --git a/CONCEPTS.md b/CONCEPTS.md index 2aafb03bf..938b73041 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -6,25 +6,71 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Provider** — an adapter plugin that connects AgentV's evaluation engine to a specific AI system (e.g., copilot CLI, copilot SDK, Claude API, pi). Each provider implements the request/response contract: given a test case, invoke the AI system and return its output. Providers are selected per-target in eval YAML and can be extended via the provider registry. -**Target** — The eval YAML or config declaration that activates a specific provider for an evaluation run. A target has a stable `id`, a `provider` backend kind, an optional `runtime`, and provider settings under `config`; field-level `file://` references can load prompts, defaults, or other config fragments at the boundary. A single eval file can declare multiple targets to compare AI systems side by side. +**Target** — The eval YAML or config declaration that activates a specific provider for an evaluation run. A target has a stable `id`, a `provider` backend kind, an optional `runtime`, and provider settings under `config`; field-level `file://` references can load prompts, defaults, or other config fragments at the boundary. A single eval file can declare multiple targets to compare AI systems side by side. Targets select agents/providers; they do not own the authored host/Docker testbed recipe. -**Provider runtime boundary** — the process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; in-process SDK providers share the orchestrator process and need either a targeted transport fix or subprocess-style isolation when runtime teardown can threaten run artifact finalization. +**Target runtime** — The placement/transport mode for invoking a target provider, such as host execution, sandbox/container placement, CLI subprocess, app-server protocol, RPC, or SDK child runner. Runtime describes how the selected agent is invoked. Advanced home/env/profile-style overlays are provider or runtime configuration details, not the authored testbed recipe. Runtime is separate from the environment that prepares files, services, and cwd. + +**Provider runtime boundary** — The process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; SDK providers should run through an AgentV-owned child-runner boundary when runtime teardown can threaten run artifact finalization. This boundary does not own repository/testbed setup by default. ## Evaluation Model **Eval / Eval YAML** — The only composable and runnable AgentV authoring primitive. An eval YAML file can be a reusable task suite that owns task context, a wrapper eval that imports suites and binds top-level runtime policy, or a sidecar around raw JSONL cases. AgentV does not have a separate runnable `experiment.yaml` artifact. -**Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `workspace`, assertions, expected references, and judge criteria. It can run directly or be imported by another eval with `tests[].include` and `type: suite`. +**Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `environment`, assertions, expected references, and judge criteria. It can run directly or be imported by another eval with `tests[].include` and `type: suite`. -**Raw case file** — YAML, JSONL, or directory case data imported with `tests: ./cases.yaml`, string shorthand, or `type: tests`. Raw cases are reusable data inputs; they do not carry imported suite context such as shared `workspace`, shared `input`, or shared `assertions`. +**Raw case file** — YAML, JSONL, or directory case data imported with `tests: ./cases.yaml`, string shorthand, or `type: tests`. Raw cases are reusable data inputs; they do not carry imported suite context such as shared `environment`, shared `input`, or shared `assertions`. -**Wrapper eval** — Eval YAML whose main job is to import task suites and bind top-level runtime policy such as target selection, repeat count, timeout, budget, and thresholds. Wrapper evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. A wrapper that imports suites with `type: suite` does not define parent `workspace`; imported suites own task environment. +**Wrapper eval** — Eval YAML whose main job is to import task suites and bind top-level runtime policy such as target selection, repeat count, timeout, budget, and thresholds. Wrapper evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. A wrapper that imports suites with `type: suite` does not define parent `environment`; imported suites own task environment unless a future scoped feature explicitly defines environment override semantics. **Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `.internal/index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `extensions` or target hooks, not in a separate experiment artifact. **Tags** — A promptfoo-shaped `Record` map authored on an eval (or project config / `--tag key=value`) that labels a run with structured facets such as `experiment`, `team`, or `env`. The reserved `experiment` key feeds the experiment namespace. The resolved map is recorded in `summary.json` `metadata.tags` and every `.internal/index.jsonl` row, and the Dashboard "Tags" tab groups and compares runs by any tag key. This is the only "tags" concept: the earlier free-form manual per-run tag chips have been removed. (Suite-level `tags` may still be authored as a string list, which is a selection construct for `select.tags` / `--tag name` filtering rather than run metadata.) -**Workspace** — The task environment an eval prepares for the agent: repositories, templates, fixture files, and post-materialization extensions. It is not prompt input; use `input` for instructions and `workspace.repos[]` for multi-repo workspaces the agent can inspect or modify through tools. `workspace.repos[]` is first-class declarative provenance, materializes before extensions run, and `workspace.scope` is `suite` or `attempt`. +**Environment** — The AgentV-authored testbed recipe for coding-agent evals. It prepares the host or Docker state an agent will inspect or modify: repositories, archives, patches, generated fixtures, services, dependency setup, and cwd. `environment` can be inline or loaded with `file://`, with shared `file://` recipes as the canonical reusable form. Initial `environment.type` values are `host` and `docker`. Promptfoo does not define this primitive; it is an AgentV extension to promptfoo-compatible eval authoring. + +**Workdir** — The current working directory inside an environment. `environment.workdir` is the cwd passed to target providers and graders/test scripts unless a later scoped feature explicitly overrides it. Host workdirs are local paths such as `./workspaces/bottle`; Docker workdirs are container paths such as `/app`. + +**Top-level `env`** — Promptfoo-compatible provider/eval environment-variable overrides and load-time template inputs such as `OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}"`. Top-level `env` is not the testbed recipe and must not be moved under `environment`. + +**`environment.env`** — Future or recipe-scoped variables for the host/Docker testbed itself, such as container process environment. It is distinct from top-level `env`, which feeds provider/eval config templating. + +**Extensions** — Promptfoo-compatible lifecycle hooks such as `beforeAll`, `beforeEach`, `afterEach`, and `afterAll`. Extensions can customize eval flow, but they are not the canonical contract for materializing repositories, Docker images, fixtures, or cwd. A `beforeAll` hook can run setup code, but then cwd, Docker/image/resource config, and testbed provenance are hidden in executable hook side effects instead of authored data. Use `environment.setup` for authored testbed setup. + +**Workspace** — Not the public AgentV coding-agent testbed contract. Where older docs or code use `workspace` or `workspace.repos[]` to mean the authored testbed, that naming is migration debt toward `environment`. The word may still describe ordinary mutable directories, local checkout state, or implementation-specific working folders when it is not a competing authored YAML primitive. + +**Environment recipe examples**: + +```yaml +environment: file://.agentv/environments/local-python.yaml + +targets: + - id: codex + provider: codex-cli +``` + +```yaml +# .agentv/environments/local-python.yaml +type: host +workdir: ./workspaces/bottle +setup: + command: ./scripts/setup-workspace.sh + args: + repo: https://github.com/bottlepy/bottle.git + commit: 0207a34f0c5716cd292dd4480253ad35d3da49f3 + path: ./workspaces/bottle +``` + +```yaml +environment: + type: docker + context: ./environment + workdir: /app + env: + NODE_ENV: test + +env: + OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}" +``` **Run bundle** — A committed local result directory at `.agentv/results//`. `summary.json` records run metadata such as `run_id` and `experiment`; `.internal/index.jsonl` records per-case rows. @@ -36,6 +82,8 @@ Shared domain vocabulary for this project — entities, named processes, and sta **Result directory** — The `result_dir` field in a `.internal/index.jsonl` row. It is a run-local directory allocation for that row's sidecars and outputs, usually a readable test-id or slug prefix plus a UUID/hash-like suffix. Consumers discover it from `.internal/index.jsonl` and must not infer it from suite names, display names, test IDs, targets, models, or folder position. +**Artifact directory** — A directory in a run bundle that stores result evidence or generated outputs. Artifact directories are discovered through fields such as `result_dir`, `grading_path`, `metrics_path`, `transcript_path`, and `outputs_path`; they are not the same thing as `environment.workdir` and do not define where the target runs. + **Artifact sidecar** — A file beside or below a result directory that provides evidence for a result, such as `summary.json`, `grading.json`, `result.json`, transcripts, logs, or outputs. Sidecars are evidence, not the primary discovery mechanism for a run. **Artifact sample folder** — A per-case `sample-N/` folder under a result directory. It stores one materialized execution's sidecars and outputs. It is not the primary comparison dimension: stochastic samples and infrastructure retries are represented with explicit `sample_index` and `retry_index` metadata rather than inferred from folder position. diff --git a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md index a4a41dd14..c02f84017 100644 --- a/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md +++ b/docs/adr/0016-promptfoo-superset-eval-authoring-contract.md @@ -22,6 +22,14 @@ describes evaluated `graders[]` and nested `checks[]` with aggregate `pass`, `passed`-only aliases, top-level `checks`, or dynamic one-grader artifact shapes as the public contract. +Status note (2026-07-05): Bead `av-noh3.2.1` supersedes this ADR's earlier +`workspace` authoring language for coding-agent testbeds. AgentV's canonical +public testbed recipe field is now `environment`, authored at suite/test/case +scope and either inline or by `file://` reference. `workspace` and +`workspace.repos` are not the canonical coding-agent benchmark contract; any +workspace-named code or docs that model the same testbed concept are migration +debt unless they refer only to internal mutable directories or result storage. + ## Context AgentV's eval-authoring surface diverged from industry primitives. We are re-basing @@ -58,7 +66,7 @@ keep AgentV's only where its semantics are genuinely better.** Dashboard can show criterion-level evidence, using the same mechanism as script graders, field accuracy, execution metrics, and tool trajectory. 3. **Grader execution**: `javascript` in-process (Bun `import`), `python` subprocess, - `script` = the subprocess power tool (workspace-`cwd`, arbitrary language). + `script` = the subprocess power tool (`environment.workdir` cwd, arbitrary language). `javascript` is NOT desugared to `script`. 4. **`metric` is the named-score field** (nunjucks-templated); grader `name` becomes display-only. Add `named_scores` + `derived_metrics`. @@ -91,21 +99,39 @@ keep AgentV's only where its semantics are genuinely better.** `repeat: { count, strategy, early_exit }` (map promptfoo `repeat:int` → `count`+`pass_all`); executable `gate` release policy (alongside per-test `threshold`); `imports`/`select`; `depends_on`. `experiment` is authored as `tags.experiment` — a plain tag with **no structural privilege** (not a bucket/field/storage path; not a privileged grouping key; tags alphabetical; default compare key is a user preference). `--experiment X` = sugar for `--tag experiment=X`. Its **value** is auto-defaulted to the eval/suite name when unset so runs are always groupable (ADR-0009 derivation) — a default value, not a privileged key (ADR-0017). -10. **Workspace repo provisioning is a declarative FIELD, not an extension.** - `workspace.repos: [{ path, repo, commit (base_commit alias), sparse?, ancestor? }]` is - declared per-test (overridable) / at suite level, and the **harness materializes it - (harness-owned resolver, ADR 0017) BEFORE any hook or the target runs.** The *common - case* is not a hand-rolled per-eval hook (ordering + reproducibility + declarative - provenance). **But acquisition is pluggable** (ADR 0017 pt5): custom acquisition is - first-class via a registered custom backend or a `beforeAll` escape hatch, and the - built-in acquisition may itself be a swappable plugin — this is the correction of an - earlier over-absolute "not an extension" claim; provenance stays a declarative field, - acquisition stays extensible. `workspace.scope` (`suite` or `attempt`) is the portable - workspace lifetime field. - **Extensions are for pluggable non-provisioning setup only**: promptfoo lifecycle - (`beforeAll`/`afterAll`/`beforeEach`/`afterEach`), running *after* materialization — - e.g. `agentv:agent-rules` (stage skills/hooks/agents) and custom `file://` hooks. - Removed: `on_run_complete`, `preprocessors` (→ `extensions`). +10. **Coding-agent testbed setup is a declarative `environment`, not a + lifecycle extension and not target identity.** AgentV remains + promptfoo-compatible where promptfoo has matching primitives: `prompts`, + `vars`, `tests`, `default_test`/`defaultTest`, `assert`, transforms, + `targets`/providers, top-level `env`, and lifecycle `extensions`. AgentV + adds `environment` as an AgentV-specific suite/test/case testbed recipe for + repo materialization, fixtures, patches, services, Docker context/image, + setup scripts, and the workdir/cwd handed to targets and graders. The + recipe may be inline or a `file://` reference; shared `file://` recipes are + the canonical reusable form: + + ```yaml + environment: file://.agentv/environments/local-python.yaml + + targets: + - id: codex + provider: codex-cli + ``` + + `environment.type` starts with `host` and `docker`. `environment.workdir` + defines the current working directory passed to target providers and + graders/test scripts unless a later scoped feature explicitly overrides it. + Top-level `env` remains promptfoo-compatible provider/eval env overrides + rendered from `{{ env.VAR }}` and must not be moved under `environment`. + If `environment.env` is implemented later, it means variables scoped to the + host/docker testbed. `environment.setup` runs scripts with typed `args` to + materialize repos, archives, patches, generated fixtures, services, or + other testbed state. + + Promptfoo `extensions` remain lifecycle hooks + (`beforeAll`/`afterAll`/`beforeEach`/`afterEach`) for customizing eval flow. + They can augment a run, but they are not the canonical testbed setup + contract. Removed: `on_run_complete`, `preprocessors` (→ `extensions`). 11. **Scope**: `similar` ships with a configured embeddings provider, `llm-rubric` ships as the free-form rubric judge, and `g-eval` covers structured or multi-criteria rubric judging. Exotic promptfoo assertions @@ -116,9 +142,10 @@ keep AgentV's only where its semantics are genuinely better.** Removed (hard): `assertions`, `composite`, `eval_cases`, `workspace.hooks` (→ `extensions`), `on_run_complete`, `preprocessors`, `${{ ENV }}`, top-level `budget_usd`, scalar top-level `threshold`, grader `name`-as-metric, the -`z.never()` rejection stubs. **Kept** as declarative fields: `workspace.repos` (provenance), -`workspace.scope`, `workspace.docker`, `workspace.template`, direct-suite -`input`, and direct-suite `input_files`. +`z.never()` rejection stubs. **Kept** as declarative fields: direct-suite +`input` and direct-suite `input_files`. **Superseded for coding-agent testbeds:** +`workspace.repos`, `workspace.scope`, `workspace.docker`, and +`workspace.template`; use the AgentV `environment` recipe contract instead. ## Consequences @@ -128,5 +155,5 @@ top-level `budget_usd`, scalar top-level `threshold`, grader `name`-as-metric, t - promptfoo authors get a near-drop-in contract (snake_case); AgentV keeps repo/agent differentiation as documented extensions. - FizzBuzz/SWE-bench-style test grading needs no new assertion primitive -- a - workspace-`cwd` `script` grader runs the tests (see ADR 0017 note on SWE-bench - `FAIL_TO_PASS`/`PASS_TO_PASS`). + script grader runs the tests from `environment.workdir` (see ADR 0017 note on + SWE-bench `FAIL_TO_PASS`/`PASS_TO_PASS`). diff --git a/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md b/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md index 55fab671f..8ace98032 100644 --- a/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md +++ b/docs/adr/0017-output-artifact-and-workspace-resolver-contract.md @@ -1,4 +1,4 @@ -# 17. Output/artifact contract + workspace resolver (provenance vs acquisition) +# 17. Output/artifact contract + environment recipe contract Date: 2026-07-02 @@ -11,11 +11,17 @@ and [ADR 0012 (finalize run artifact layout)](0012-finalize-run-artifact-layout. extends [ADR 0008 (normalized transcript)](0008-normalized-transcript-artifact-contract.md). Companion to [ADR 0016](0016-promptfoo-superset-eval-authoring-contract.md). +Amended (2026-07-05) by Bead `av-noh3.2.1`: the output/artifact contract remains +accepted, but the earlier workspace resolver naming and `workspace.repos` +authoring shape are superseded for coding-agent testbeds. AgentV now uses +`environment` as the public suite/test/case testbed recipe. `workspace` is no +longer a locked canonical public testbed term. + ## Context We reviewed the output formats of promptfoo, margin-lab, vercel-agent-eval, and -agentskills, and the workspace-acquisition models of SWE-bench, margin, Harbor, and -Inspect AI (`docs/plans/…` §6, §11.1). The active implementation scope is tracked in +agentskills, and the testbed/acquisition models of SWE-bench, margin, Harbor, +and Inspect AI (`docs/plans/…` §6, §11.1). The active implementation scope is tracked in Beads; PRs, ADRs, and plans summarize those Beads for review, but the Bead descriptions, acceptance criteria, and notes remain the implementation source of truth. @@ -29,7 +35,8 @@ full `gradingResult`; `GradingResult` carries `pass`, `score`, `reason`, optiona results, prompts, stats, config, author, and variables. AgentV borrows the clean aggregate grading vocabulary and component-breakdown idea, then writes them into a filesystem/Git-native split run bundle instead of one consolidated DB/export file. -Two decisions follow: the canonical result bundle, and how a workspace is acquired. +Two decisions follow: the canonical result bundle, and how a coding-agent +environment recipe is authored and materialized. ## Decision — output/artifact contract (best-of-each, split, no DB) @@ -173,77 +180,132 @@ and `outputs_path`, not a full embedded grading tree. - **Repeat folder = `sample-N`, not `run-N`.** "run" is overloaded (`run_id` = the whole invocation). Rename `run-${attempt+1}` → `sample-1`, `sample-2`, … (matches margin `samples_per_case`/`sample_index`, explicit sampling metrics, and AgentV's `repeat`; Inspect's `epoch` is the ML-jargon alt). Keep the metadata split: `sample_index` = repeats, `retry_index` = infra retries. - **`experiment` has no *structural* privilege, but its *value* is auto-defaulted.** No storage dir (already `/`), no top-level field (`tags.experiment`), no special schema; tag keys sort **alphabetically**; the default grouping/compare **key** is a user preference (any tag — AgentV blesses none). `--experiment X` = sugar for `--tag experiment=X`. **The one convenience:** the harness auto-populates the `experiment` tag's **value** when unset, deriving it from the eval/suite name (ADR-0009: `--experiment` > authored `tags.experiment` > eval/suite name). So every run always has a meaningful `experiment` value and is groupable — without the author setting anything. This is a default *value*, not a privileged *key*. -## Decision — workspace resolver (provenance vs acquisition) - -Cross-framework convergent rule (SWE-bench, Terminal-bench, margin, lm-eval, Inspect): -**the case declares WHAT (identity + pin); the harness resolves WHERE-FROM via a -selectable backend. Nobody puts acquisition in the task.** - -**Field (WHAT) and resolver (HOW) are orthogonal — both required, neither replaces the -other.** Analogy: `package.json` vs the package registry. `dependencies: {lodash: ^4}` is -the **field** (always declared); npm's registry/mirror/tarball resolution is **pluggable -acquisition** — you can point npm at a custom registry, but you don't delete `package.json`. -Likewise: the `workspace` field declares provenance; the resolver (built-in backends + -custom-backend plugin + `beforeAll` escape hatch) is the pluggable *how*. A custom backend -still reads the field to know which `repo`+`commit` to fetch. You only "don't need the -field" if you go full escape-hatch and forgo declarative provenance (not recommended). - -### Naming: `workspace` (durable, locked) -Chosen over alternatives for longevity — it names the *what* (a working directory), not the -*how*: CI-standard (`GITHUB_WORKSPACE`), used by margin-lab, git/Cargo/Bazel/VS Code. -Rejected: `sandbox` (Inspect — connotes an isolation boundary, which is a *property* → the -`isolation`/`docker` fields, not the concept); `environment` (overloaded with env vars); -`testbed` (SWE-bench jargon). - -### Final locked schema +## Decision — environment recipe contract + +Cross-framework evidence from Bead `av-noh3.1` supports a different public +contract than this ADR's original `workspace` decision: + +- Promptfoo local clone `/home/entity/projects/promptfoo/promptfoo` at commit + `6bfc5a0c7f16f9c4717ac731d276b578e63d0769` has top-level `env` for + provider/eval env overrides and `extensions` for lifecycle hooks; it has no + typed testbed/environment primitive. +- Harbor local clone `/home/entity/projects/harbor-framework/harbor` at commit + `a9148a9509a0bc0cbeb80375aa619bd5cdb5845c` models task environments with + Docker image/resources/env/workdir and keeps agent setup separate from + environment start. +- Terminal-Bench 2 clone `/tmp/agentv-terminal-bench-2` at commit + `2fd12b88aafdd04a52c298e3940bcb189f9766d6` uses `task.toml`, + `environment/Dockerfile`, `tests/test.sh`, and `solution/solve.sh`; Docker + tasks commonly prepare `/app` by cloning/pinning repositories or copying + challenge files. +- Margin clone `/home/entity/projects/Margin-Lab/evals` at commit + `53fb2fd080689efaf7934573d8759d14fc1043e4` keeps case image/cwd separate + from selected agent configuration. + +AgentV combines promptfoo-compatible eval authoring with +Harbor/Terminal-Bench/Margin-style coding-agent environments: + +```yaml +environment: file://.agentv/environments/local-python.yaml + +targets: + - id: codex + provider: codex-cli +``` + ```yaml -workspace: # suite-level default; tests[].workspace overrides per case - repos: # PROVENANCE only (what to materialize) - - path: ./CargoWise # where it lands in the workspace - repo: https://github.com/WiseTechGlobal/CargoWise.git # canonical identity (join key) - commit: 953adb9 # immutable SHA pin (base_commit accepted as input alias) - sparse: [src/X] # optional content selection - ancestor: 1 # optional (nth-ancestor pin) - scope: attempt # suite (default) | attempt - template: ./tmpl # optional local scaffold - docker: { image: ... } # optional container env +# .agentv/environments/local-python.yaml +type: host +workdir: ./workspaces/bottle +setup: + command: ./scripts/setup-workspace.sh + args: + repo: https://github.com/bottlepy/bottle.git + commit: 0207a34f0c5716cd292dd4480253ad35d3da49f3 + path: ./workspaces/bottle ``` -**Never in this schema:** acquisition (resolver + backends → harness/machine config, keyed -on `repo`) and hooks (→ `extensions`). Keeping those out is what makes the schema durable — -new acquisition technology plugs in without touching it. `commit` is an immutable SHA -(reproducible); mutable refs are excluded. - -1. **Eval declares provenance ONLY, in a declarative `workspace.repos` field** (per-test - overridable / suite-level; NOT a `vars` blob and NOT an extension): `workspace.repos: - [{ path, repo, commit (base_commit alias), sparse?, ancestor? }]`, plus `workspace.scope` - (`suite` or `attempt`). Remove the tangled acquisition fields (`type`/`local`, `resolve`, - `clone.depth`, `clone.filter`, per-repo `resolver`). The harness materializes this - **before hooks** (ADR 0016 pt10). -2. **Acquisition = harness resolver in machine config (`$AGENTV_HOME/config.yaml`), - keyed on `repo`**, ordered backends: (1) local checkout auto-adopt via origin-match - → mirror cache; (2) configured local mirror; (3) custom command resolver returning - a flat `{status,path}` acquisition source, including Git sources or static directory - snapshots; (4) AgentV mirror cache and remote clone; - (5) *future* Docker image (SWE-bench/margin/Inspect — same identity key, new backend; - adopt Inspect's `image`/`build`/`x-local` distinction + per-config init caching). -3. **`--reference` (mirror cache) is the workhorse**: shallow-speed WITH full history, so - deep `base_commit` pins never break — retires the `--depth`/`--filter` debate. Keep - `sparse` for content selection. -4. **Materialization runs before hooks and reads the declared provenance**; ordinary user - `beforeAll`/`beforeEach` hooks run *after* it. Resolver config is machine-local, - orthogonal to eval and target YAML. Targets carry no repos. -5. **The resolver is PLUGGABLE — custom acquisition is first-class** (per the "plugins over - built-ins" product guardrail). Two extension points beyond the built-in backends: - (a) **register a custom acquisition backend** (a resolver plugin, config-level, keyed on - `repo`) for a bespoke store/format — the recommended path; (b) a **`beforeAll` extension - escape hatch** that materializes a fully author-owned workspace and reports its path - (what the promptfoo parity example did). The built-in acquisition itself may be - implemented as an auto-registered, ordered-first, **swappable** plugin over the same - public interface — so the default is zero-config yet replaceable. - -The invariants (not the mechanism) are what matter: provenance is declared as data; -acquisition runs before hooks and is keyed on the pin; built-ins ship. New backends — -built-in or user — plug in without touching the eval schema because all resolve the same pin. + +```yaml +environment: + type: docker + context: ./environment + workdir: /app + env: + NODE_ENV: test + +env: + OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}" +``` + +### Why not Promptfoo `extensions` as the testbed contract? + +Promptfoo-compatible extensions remain supported lifecycle hooks, and they can +mechanically run setup code. For example, an eval could author a reusable +`beforeAll` setup hook: + +```yaml +extensions: + - file://.agentv/extensions/setup-workspace.ts:beforeAll + +targets: + - id: codex + provider: codex-cli +``` + +That shape is not a good canonical product contract for coding-agent eval +testbeds. The working directory becomes an implicit side effect of hook code +rather than a typed field in the authored suite/test/case. The target has no +obvious typed place to receive cwd after the hook runs. Docker image, build +context, resource, mount, and service configuration disappear into executable +hook code instead of staying visible as reviewable data. Multiple suites can +share the hook file, but reviewers and coding agents still cannot quickly see +the testbed from the YAML that defines the benchmark case. It also diverges +from Harbor, Margin, and Terminal-Bench-style explicit task +environment/testbed contracts, where task substrate and agent selection are +separate visible concepts. + +Therefore `extensions` stay lifecycle hooks for customizing eval flow, while +`environment` is the explicit AgentV substrate, setup, and `workdir` contract. + +1. **`environment` is the authored testbed recipe** at suite/test/case scope. + It may be inline or loaded through a field-level `file://` reference. Shared + `file://` recipes are the canonical reusable form. +2. **Initial recipe types are `host` and `docker`.** `host` prepares a local + trusted-machine workdir. `docker` prepares a container-backed testbed using + fields such as `context`, `dockerfile`, `image`, `workdir`, and future scoped + resource/mount/secrets fields. +3. **`environment.workdir` defines cwd.** AgentV passes the resolved workdir to + target providers and graders/test scripts unless a later scoped feature + explicitly overrides it. Target configs may still expose provider-specific + knobs, but the canonical testbed cwd comes from the environment recipe. +4. **`environment.setup` materializes testbed state.** Setup is declarative data + plus a command and typed `args`: repos, archives, patches, generated + fixtures, installed dependencies, services, and other case state. Setup runs + before target execution and before ordinary promptfoo lifecycle hooks. +5. **Top-level `env` remains promptfoo-compatible.** It is for provider/eval env + overrides and load-time `{{ env.VAR }}` rendering. Do not move it under + `environment`. If `environment.env` is implemented, it means variables scoped + to the host/docker testbed, not promptfoo env templating. +6. **Promptfoo `extensions` remain lifecycle hooks.** They can customize eval + flow, but they are not the canonical testbed setup contract because hidden + hook code is weaker for review, validation, sharing, and cwd semantics. +7. **Targets select agents/providers.** `targets[].id` is stable AgentV target + identity, `targets[].provider` names the adapter/control boundary, and + `targets[].runtime` remains placement/transport. Targets do not own + Docker/testbed setup by default. +8. **`workspace` is not the public coding-agent benchmark contract.** The + original `workspace.repos`, `workspace.scope`, `workspace.docker`, and + `workspace.template` names are superseded where they meant authored testbed + setup. Existing workspace-named code is migration debt when it models this + same concept. The word may still appear for unrelated internal mutable + directories, caches, or result/artifact paths, but not as a competing + authored testbed primitive. + +The invariants matter more than the mechanism: testbed setup is declared as +data; materialization precedes target execution and normal lifecycle hooks; cwd +is explicit; provider/target identity remains separate from testbed setup; and +run bundles can snapshot the resolved recipe, setup inputs, and resolved +workdir as provenance. ### Note: SWE-bench `FAIL_TO_PASS` / `PASS_TO_PASS` @@ -251,15 +313,15 @@ built-in or user — plug in without touching the eval schema because all resolv dataset row. The distinction (fix-tests vs regression-tests) matters only at *dataset-construction* time; at *run* time it collapses to "**run these named tests; pass iff all pass**". So it is **too domain-specific for a core primitive, and needs no -dedicated SDK recipe** -- it is plainly a workspace-`cwd` **`script`** grader: the grader runs -the repo's tests in the workspace and its exit code is the verdict (exactly margin's -`tests/test.sh` 0/1/2 model). The two lists are just data the grader's command consumes -(inline, or from `vars`/`metadata`). Combined with the Docker-image acquisition backend -(#5), this is how AgentV runs SWE-bench natively — same `repo`+`commit` provenance, no -schema change, no new grader type. +dedicated SDK recipe** -- it is plainly a **`script`** grader run from +`environment.workdir`: the grader runs the repo's tests and its exit code is the +verdict (exactly margin's `tests/test.sh` 0/1/2 model). The two lists are just +data the grader's command consumes (inline, or from `vars`/`metadata`). Combined +with Docker environment recipes, this is how AgentV runs SWE-bench-style tasks +natively: explicit testbed recipe, explicit cwd, no new grader type. ### Cross-check: exploitbench (confirms + two borrowables) -exploitbench (security-exploit benchmark; AgentV research `entities/exploitbench.md`) **confirms** this contract: split filesystem run-tree is the source of truth (`job.json`/`score.json`/`cost.json`/`transcript.jsonl`/`tool_calls.jsonl`/`config_snapshot.yaml`); its SQLite is a **derived, rebuildable view** (`import`/`export` bijection), not required — validating our no-DB core (jq + `index.jsonl` is the query surface; a SQLite view stays an optional post-run adapter, Phoenix boundary intact). Docker images are pinned by `sha256:` digest at run start (reinforces resolver backend #5); `config_snapshot` = our `bundle.json`. **Borrow:** (1) a **`provenance`** field on result rows (`native`/`mock`/`replay`/`imported_from_*`) — durable, fits AgentV's replay/transcript/mock providers; adopt now. (2) **Eval-integrity / anti-reward-hacking — future scope**: run high-stakes graders in a fresh container with the workspace mounted **read-only**; an `audit` pass that re-grades from the stored transcript, scans for reward-hacking red flags, and verifies model identity (the provider served the requested model). "Post-hoc audit as part of benchmark validity." +exploitbench (security-exploit benchmark; AgentV research `entities/exploitbench.md`) **confirms** this contract: split filesystem run-tree is the source of truth (`job.json`/`score.json`/`cost.json`/`transcript.jsonl`/`tool_calls.jsonl`/`config_snapshot.yaml`); its SQLite is a **derived, rebuildable view** (`import`/`export` bijection), not required — validating our no-DB core (jq + `index.jsonl` is the query surface; a SQLite view stays an optional post-run adapter, Phoenix boundary intact). Docker images are pinned by `sha256:` digest at run start; `config_snapshot` = our `bundle.json`. **Borrow:** (1) a **`provenance`** field on result rows (`native`/`mock`/`replay`/`imported_from_*`) — durable, fits AgentV's replay/transcript/mock providers; adopt now. (2) **Eval-integrity / anti-reward-hacking — future scope**: run high-stakes graders in a fresh container with the prepared environment mounted **read-only**; an `audit` pass that re-grades from the stored transcript, scans for reward-hacking red flags, and verifies model identity (the provider served the requested model). "Post-hoc audit as part of benchmark validity." ## Consequences @@ -267,6 +329,7 @@ exploitbench (security-exploit benchmark; AgentV research `entities/exploitbench 0011/0012 marked accordingly. - Opik export (`av-bv4.6`) and the Dashboard (`av-2s7`) consume the new bundle → re-gate on this contract. -- Codemod handles bundle-field renames and drops the tangled repo-acquisition fields. +- Codemod handles bundle-field renames and drops the superseded workspace + repo-acquisition fields where they modeled authored testbed setup. - `camelCase` in an AgentV-owned artifact or response is a contract bug, not a stylistic alternative. diff --git a/docs/adr/0018-coding-agent-target-runtime-contract.md b/docs/adr/0018-coding-agent-target-runtime-contract.md index b529eb8fd..63267b587 100644 --- a/docs/adr/0018-coding-agent-target-runtime-contract.md +++ b/docs/adr/0018-coding-agent-target-runtime-contract.md @@ -13,6 +13,14 @@ Profile and sandbox isolation evidence is deferred to `av-t2o5.1`. Provider-agnostic recorded trajectory replay and removal of `provider: copilot-log` from authored target YAML are owned by `av-t2o5.2`. +Amended (2026-07-05) by Bead `av-noh3.2.1`: AgentV's coding-agent testbed +recipe is `environment` at suite/test/case scope, not a target field. This ADR +continues to reject target-level `environment`, `container`, and install fields; +targets receive the resolved `environment.workdir` as cwd unless a later scoped +feature explicitly overrides it. Runtime mode vocabulary may be refined by +Bead `av-oi9a`; that separate cleanup does not move testbed setup under +targets. + ## Context AgentV evaluates coding agents in real repositories. Those agents are not @@ -94,6 +102,14 @@ under `evaluate_options.max_concurrency`, not inside a target definition. Grader selection belongs to `defaults.grader`, CLI overrides, or evaluator-level target selection, not to the system-under-test target. +`environment` is valid as an AgentV eval/test/case recipe field outside target +definitions. It prepares the host or Docker testbed and defines +`environment.workdir`; the target runtime then decides how the selected +provider/agent is invoked against that prepared cwd. `environment` does not +replace `targets[].id`, `targets[].provider`, or `targets[].runtime`. +Docker/testbed setup belongs to the environment driver by default, not to each +target provider. + ### Provider Boundaries Process and protocol providers are the preferred defaults: