docs(adr): codify environment recipes for coding-agent evals

CONCEPTS.md

@@ -6,25 +6,71 @@ Shared domain vocabulary for this project — entities, named processes, and sta
 
 **Provider** — an adapter plugin that connects AgentV's evaluation engine to a specific AI system (e.g., copilot CLI, copilot SDK, Claude API, pi). Each provider implements the request/response contract: given a test case, invoke the AI system and return its output. Providers are selected per-target in eval YAML and can be extended via the provider registry.
 
-**Target** — The eval YAML or config declaration that activates a specific provider for an evaluation run. A target has a stable `id`, a `provider` backend kind, an optional `runtime`, and provider settings under `config`; field-level `file://` references can load prompts, defaults, or other config fragments at the boundary. A single eval file can declare multiple targets to compare AI systems side by side.
+**Target** — The eval YAML or config declaration that activates a specific provider for an evaluation run. A target has a stable `id`, a `provider` backend kind, an optional `runtime`, and provider settings under `config`; field-level `file://` references can load prompts, defaults, or other config fragments at the boundary. A single eval file can declare multiple targets to compare AI systems side by side. Targets select agents/providers; they do not own the authored host/Docker testbed recipe.
 
-**Provider runtime boundary** — the process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; in-process SDK providers share the orchestrator process and need either a targeted transport fix or subprocess-style isolation when runtime teardown can threaten run artifact finalization.
+**Target runtime** — The placement/transport mode for invoking a target provider, such as host execution, sandbox/container placement, CLI subprocess, app-server protocol, RPC, or SDK child runner. Runtime describes how the selected agent is invoked. Advanced home/env/profile-style overlays are provider or runtime configuration details, not the authored testbed recipe. Runtime is separate from the environment that prepares files, services, and cwd.
+
+**Provider runtime boundary** — The process boundary between AgentV's evaluation orchestrator and the agent runtime a provider invokes. CLI-backed providers place the agent runtime outside the orchestrator; SDK providers should run through an AgentV-owned child-runner boundary when runtime teardown can threaten run artifact finalization. This boundary does not own repository/testbed setup by default.
 
 ## Evaluation Model
 
 **Eval / Eval YAML** — The only composable and runnable AgentV authoring primitive. An eval YAML file can be a reusable task suite that owns task context, a wrapper eval that imports suites and binds top-level runtime policy, or a sidecar around raw JSONL cases. AgentV does not have a separate runnable `experiment.yaml` artifact.
 
-**Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `workspace`, assertions, expected references, and judge criteria. It can run directly or be imported by another eval with `tests[].include` and `type: suite`.
+**Task suite** — Eval YAML that owns what is being tested: prompts, datasets, input files, fixtures, `environment`, assertions, expected references, and judge criteria. It can run directly or be imported by another eval with `tests[].include` and `type: suite`.
 
-**Raw case file** — YAML, JSONL, or directory case data imported with `tests: ./cases.yaml`, string shorthand, or `type: tests`. Raw cases are reusable data inputs; they do not carry imported suite context such as shared `workspace`, shared `input`, or shared `assertions`.
+**Raw case file** — YAML, JSONL, or directory case data imported with `tests: ./cases.yaml`, string shorthand, or `type: tests`. Raw cases are reusable data inputs; they do not carry imported suite context such as shared `environment`, shared `input`, or shared `assertions`.
 
-**Wrapper eval** — Eval YAML whose main job is to import task suites and bind top-level runtime policy such as target selection, repeat count, timeout, budget, and thresholds. Wrapper evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. A wrapper that imports suites with `type: suite` does not define parent `workspace`; imported suites own task environment.
+**Wrapper eval** — Eval YAML whose main job is to import task suites and bind top-level runtime policy such as target selection, repeat count, timeout, budget, and thresholds. Wrapper evals may live under an `experiments/` directory, but that path is an optional user-owned convention and AgentV does not infer behavior from it. A wrapper that imports suites with `type: suite` does not define parent `environment`; imported suites own task environment unless a future scoped feature explicitly defines environment override semantics.
 
 **Experiment** — A string metadata/run-grouping label such as `baseline`, `candidate`, `with_skills`, or `without_skills`. It is not a runtime-policy object and not a result path namespace. Experiment is expressed as the reserved `tags.experiment` key (see **Tags**); there is no top-level `experiment` field. Runtime policy belongs in top-level eval fields or target objects; the experiment label is recorded in `summary.json` and `.internal/index.jsonl` for Dashboard grouping and comparison. Lifecycle setup belongs in `extensions` or target hooks, not in a separate experiment artifact.
 
 **Tags** — A promptfoo-shaped `Record<string,string>` map authored on an eval (or project config / `--tag key=value`) that labels a run with structured facets such as `experiment`, `team`, or `env`. The reserved `experiment` key feeds the experiment namespace. The resolved map is recorded in `summary.json` `metadata.tags` and every `.internal/index.jsonl` row, and the Dashboard "Tags" tab groups and compares runs by any tag key. This is the only "tags" concept: the earlier free-form manual per-run tag chips have been removed. (Suite-level `tags` may still be authored as a string list, which is a selection construct for `select.tags` / `--tag name` filtering rather than run metadata.)
 
-**Workspace** — The task environment an eval prepares for the agent: repositories, templates, fixture files, and post-materialization extensions. It is not prompt input; use `input` for instructions and `workspace.repos[]` for multi-repo workspaces the agent can inspect or modify through tools. `workspace.repos[]` is first-class declarative provenance, materializes before extensions run, and `workspace.scope` is `suite` or `attempt`.
+**Environment** — The AgentV-authored testbed recipe for coding-agent evals. It prepares the host or Docker state an agent will inspect or modify: repositories, archives, patches, generated fixtures, services, dependency setup, and cwd. `environment` can be inline or loaded with `file://`, with shared `file://` recipes as the canonical reusable form. Initial `environment.type` values are `host` and `docker`. Promptfoo does not define this primitive; it is an AgentV extension to promptfoo-compatible eval authoring.
+
+**Workdir** — The current working directory inside an environment. `environment.workdir` is the cwd passed to target providers and graders/test scripts unless a later scoped feature explicitly overrides it. Host workdirs are local paths such as `./workspaces/bottle`; Docker workdirs are container paths such as `/app`.
+
+**Top-level `env`** — Promptfoo-compatible provider/eval environment-variable overrides and load-time template inputs such as `OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}"`. Top-level `env` is not the testbed recipe and must not be moved under `environment`.
+
+**`environment.env`** — Future or recipe-scoped variables for the host/Docker testbed itself, such as container process environment. It is distinct from top-level `env`, which feeds provider/eval config templating.
+
+**Extensions** — Promptfoo-compatible lifecycle hooks such as `beforeAll`, `beforeEach`, `afterEach`, and `afterAll`. Extensions can customize eval flow, but they are not the canonical contract for materializing repositories, Docker images, fixtures, or cwd. A `beforeAll` hook can run setup code, but then cwd, Docker/image/resource config, and testbed provenance are hidden in executable hook side effects instead of authored data. Use `environment.setup` for authored testbed setup.
+
+**Workspace** — Not the public AgentV coding-agent testbed contract. Where older docs or code use `workspace` or `workspace.repos[]` to mean the authored testbed, that naming is migration debt toward `environment`. The word may still describe ordinary mutable directories, local checkout state, or implementation-specific working folders when it is not a competing authored YAML primitive.
+
+**Environment recipe examples**:
+
+```yaml
+environment: file://.agentv/environments/local-python.yaml
+
+targets:
+  - id: codex
+    provider: codex-cli
+```
+
+```yaml
+# .agentv/environments/local-python.yaml
+type: host
+workdir: ./workspaces/bottle
+setup:
+  command: ./scripts/setup-workspace.sh
+  args:
+    repo: https://github.com/bottlepy/bottle.git
+    commit: 0207a34f0c5716cd292dd4480253ad35d3da49f3
+    path: ./workspaces/bottle
+```
+
+```yaml
+environment:
+  type: docker
+  context: ./environment
+  workdir: /app
+  env:
+    NODE_ENV: test
+
+env:
+  OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}"
+```
 
 **Run bundle** — A committed local result directory at `.agentv/results/<run_id>/`. `summary.json` records run metadata such as `run_id` and `experiment`; `.internal/index.jsonl` records per-case rows.
 
@@ -36,6 +82,8 @@ Shared domain vocabulary for this project — entities, named processes, and sta
 
 **Result directory** — The `result_dir` field in a `.internal/index.jsonl` row. It is a run-local directory allocation for that row's sidecars and outputs, usually a readable test-id or slug prefix plus a UUID/hash-like suffix. Consumers discover it from `.internal/index.jsonl` and must not infer it from suite names, display names, test IDs, targets, models, or folder position.
 
+**Artifact directory** — A directory in a run bundle that stores result evidence or generated outputs. Artifact directories are discovered through fields such as `result_dir`, `grading_path`, `metrics_path`, `transcript_path`, and `outputs_path`; they are not the same thing as `environment.workdir` and do not define where the target runs.
+
 **Artifact sidecar** — A file beside or below a result directory that provides evidence for a result, such as `summary.json`, `grading.json`, `result.json`, transcripts, logs, or outputs. Sidecars are evidence, not the primary discovery mechanism for a run.
 
 **Artifact sample folder** — A per-case `sample-N/` folder under a result directory. It stores one materialized execution's sidecars and outputs. It is not the primary comparison dimension: stochastic samples and infrastructure retries are represented with explicit `sample_index` and `retry_index` metadata rather than inferred from folder position.

docs/adr/0016-promptfoo-superset-eval-authoring-contract.md

@@ -22,6 +22,14 @@ describes evaluated `graders[]` and nested `checks[]` with aggregate `pass`,
 `passed`-only aliases, top-level `checks`, or dynamic one-grader artifact shapes
 as the public contract.
 
+Status note (2026-07-05): Bead `av-noh3.2.1` supersedes this ADR's earlier
+`workspace` authoring language for coding-agent testbeds. AgentV's canonical
+public testbed recipe field is now `environment`, authored at suite/test/case
+scope and either inline or by `file://` reference. `workspace` and
+`workspace.repos` are not the canonical coding-agent benchmark contract; any
+workspace-named code or docs that model the same testbed concept are migration
+debt unless they refer only to internal mutable directories or result storage.
+
 ## Context
 
 AgentV's eval-authoring surface diverged from industry primitives. We are re-basing
@@ -58,7 +66,7 @@ keep AgentV's only where its semantics are genuinely better.**
    Dashboard can show criterion-level evidence, using the same mechanism as
    script graders, field accuracy, execution metrics, and tool trajectory.
 3. **Grader execution**: `javascript` in-process (Bun `import`), `python` subprocess,
-   `script` = the subprocess power tool (workspace-`cwd`, arbitrary language).
+   `script` = the subprocess power tool (`environment.workdir` cwd, arbitrary language).
    `javascript` is NOT desugared to `script`.
 4. **`metric` is the named-score field** (nunjucks-templated); grader `name` becomes
    display-only. Add `named_scores` + `derived_metrics`.
@@ -91,21 +99,39 @@ keep AgentV's only where its semantics are genuinely better.**
    `repeat: { count, strategy, early_exit }` (map promptfoo
    `repeat:int` → `count`+`pass_all`); executable `gate` release policy (alongside per-test
    `threshold`); `imports`/`select`; `depends_on`. `experiment` is authored as `tags.experiment` — a plain tag with **no structural privilege** (not a bucket/field/storage path; not a privileged grouping key; tags alphabetical; default compare key is a user preference). `--experiment X` = sugar for `--tag experiment=X`. Its **value** is auto-defaulted to the eval/suite name when unset so runs are always groupable (ADR-0009 derivation) — a default value, not a privileged key (ADR-0017).
-10. **Workspace repo provisioning is a declarative FIELD, not an extension.**
-    `workspace.repos: [{ path, repo, commit (base_commit alias), sparse?, ancestor? }]` is
-    declared per-test (overridable) / at suite level, and the **harness materializes it
-    (harness-owned resolver, ADR 0017) BEFORE any hook or the target runs.** The *common
-    case* is not a hand-rolled per-eval hook (ordering + reproducibility + declarative
-    provenance). **But acquisition is pluggable** (ADR 0017 pt5): custom acquisition is
-    first-class via a registered custom backend or a `beforeAll` escape hatch, and the
-    built-in acquisition may itself be a swappable plugin — this is the correction of an
-    earlier over-absolute "not an extension" claim; provenance stays a declarative field,
-    acquisition stays extensible. `workspace.scope` (`suite` or `attempt`) is the portable
-    workspace lifetime field.
-    **Extensions are for pluggable non-provisioning setup only**: promptfoo lifecycle
-    (`beforeAll`/`afterAll`/`beforeEach`/`afterEach`), running *after* materialization —
-    e.g. `agentv:agent-rules` (stage skills/hooks/agents) and custom `file://` hooks.
-    Removed: `on_run_complete`, `preprocessors` (→ `extensions`).
+10. **Coding-agent testbed setup is a declarative `environment`, not a
+    lifecycle extension and not target identity.** AgentV remains
+    promptfoo-compatible where promptfoo has matching primitives: `prompts`,
+    `vars`, `tests`, `default_test`/`defaultTest`, `assert`, transforms,
+    `targets`/providers, top-level `env`, and lifecycle `extensions`. AgentV
+    adds `environment` as an AgentV-specific suite/test/case testbed recipe for
+    repo materialization, fixtures, patches, services, Docker context/image,
+    setup scripts, and the workdir/cwd handed to targets and graders. The
+    recipe may be inline or a `file://` reference; shared `file://` recipes are
+    the canonical reusable form:
+
+    ```yaml
+    environment: file://.agentv/environments/local-python.yaml
+
+    targets:
+      - id: codex
+        provider: codex-cli
+    ```
+
+    `environment.type` starts with `host` and `docker`. `environment.workdir`
+    defines the current working directory passed to target providers and
+    graders/test scripts unless a later scoped feature explicitly overrides it.
+    Top-level `env` remains promptfoo-compatible provider/eval env overrides
+    rendered from `{{ env.VAR }}` and must not be moved under `environment`.
+    If `environment.env` is implemented later, it means variables scoped to the
+    host/docker testbed. `environment.setup` runs scripts with typed `args` to
+    materialize repos, archives, patches, generated fixtures, services, or
+    other testbed state.
+
+    Promptfoo `extensions` remain lifecycle hooks
+    (`beforeAll`/`afterAll`/`beforeEach`/`afterEach`) for customizing eval flow.
+    They can augment a run, but they are not the canonical testbed setup
+    contract. Removed: `on_run_complete`, `preprocessors` (→ `extensions`).
 11. **Scope**: `similar` ships with a configured embeddings provider, `llm-rubric` ships
     as the free-form rubric judge, and `g-eval` covers structured or multi-criteria
     rubric judging. Exotic promptfoo assertions
@@ -116,9 +142,10 @@ keep AgentV's only where its semantics are genuinely better.**
 Removed (hard): `assertions`, `composite`, `eval_cases`,
 `workspace.hooks` (→ `extensions`), `on_run_complete`, `preprocessors`, `${{ ENV }}`,
 top-level `budget_usd`, scalar top-level `threshold`, grader `name`-as-metric, the
-`z.never()` rejection stubs. **Kept** as declarative fields: `workspace.repos` (provenance),
-`workspace.scope`, `workspace.docker`, `workspace.template`, direct-suite
-`input`, and direct-suite `input_files`.
+`z.never()` rejection stubs. **Kept** as declarative fields: direct-suite
+`input` and direct-suite `input_files`. **Superseded for coding-agent testbeds:**
+`workspace.repos`, `workspace.scope`, `workspace.docker`, and
+`workspace.template`; use the AgentV `environment` recipe contract instead.
 
 ## Consequences
 
@@ -128,5 +155,5 @@ top-level `budget_usd`, scalar top-level `threshold`, grader `name`-as-metric, t
 - promptfoo authors get a near-drop-in contract (snake_case); AgentV keeps repo/agent
   differentiation as documented extensions.
 - FizzBuzz/SWE-bench-style test grading needs no new assertion primitive -- a
-  workspace-`cwd` `script` grader runs the tests (see ADR 0017 note on SWE-bench
-  `FAIL_TO_PASS`/`PASS_TO_PASS`).
+  script grader runs the tests from `environment.workdir` (see ADR 0017 note on
+  SWE-bench `FAIL_TO_PASS`/`PASS_TO_PASS`).