Skip to content

fix(campaign): isolate proposer selection from test#355

Draft
drewstone wants to merge 1 commit into
mainfrom
fix/proposer-split-contamination
Draft

fix(campaign): isolate proposer selection from test#355
drewstone wants to merge 1 commit into
mainfrom
fix/proposer-split-contamination

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

What changed

  • make compareProposers own three non-empty, pairwise-disjoint partitions: train, selection, and untouched test
  • pass only train and selection through ProposerOptimizationData; finish every optimizer before the first test dispatch
  • migrate GEPA, SkillOpt, FAPO, AppWorld, GSM8K, and the canonical comparison example to selection-based candidate acceptance
  • fail closed on the ambiguous legacy holdoutScenarios API and rename result fields to reflect what was actually measured
  • mark the checked-in June proposer result as selection-contaminated instead of preserving an invalid research claim

Why

SkillOpt reused holdoutScenarios adaptively for edit acceptance, then compareProposers ranked every method on those same rows. That makes the final ranking optimistically biased. The new API makes candidate selection and final testing separate by construction and rejects overlap before any scoring.

Compatibility

This is intentionally breaking rather than silently treating selection data as test data. Callers must provide trainScenarios, selectionScenarios, and testScenarios; runSkillOpt callers must rename adaptive holdout data to selectionScenarios.

Verification

  • pnpm typecheck
  • pnpm vitest run tests/campaign/compare-proposers.test.ts tests/campaign/skill-opt.test.ts (40 passed)
  • pnpm test (2,942 passed, 2 skipped)
  • pnpm build
  • pnpm lint (passes with four pre-existing warnings outside this diff)
  • pnpm verify:package
  • git merge-tree --write-tree origin/main HEAD

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant