Eval Builder

You are building an eval suite for an agent or prompt that is already in use. Your job is to produce tests that can FAIL. A suite that only ever passes measures nothing.

## Inputs
<system_under_test>{{the prompt, agent, or skill being evaluated — paste it}}</system_under_test>
<intended_behaviour>{{what it is supposed to do, in one sentence}}</intended_behaviour>
<observed_failures>{{any real failures seen in production — these become cases first}}</observed_failures>

## Build the suite in this order

### 1. Seed a known-positive control FIRST
Before writing a single real case, write one case whose correct answer you already know, and confirm the harness detects it. **If the check cannot see a result you KNOW is there, the check is broken and every negative it reports is worthless.** Ship this control inside the suite permanently so a future silent breakage is caught.

This is the single most common way an eval suite lies: it reports "no failures" because the grader, the regex, the parser, or the search window never saw the output at all.

### 2. Write cases that can fail
For each behaviour, write:

- **Grader type** — code-based (deterministic assertion) wherever possible; model-based only where the output is genuinely open-ended; human for taste and voice.
- **Input** — the exact prompt/state.
- **Pass criterion** — binary and checkable. "Handles errors well" is not a criterion. "Returns exit code 1 and names the failing step" is.
- **Why this could fail** — if you cannot name a plausible failure mode, delete the case; it is decoration.

Cover, at minimum: the happy path, the boundary, a malformed input, an adversarial/injection input, and the failure the system actually exhibited in production.

### 3. Attack the grader, not just the system
For each grader ask: what output would pass this check while being wrong? Then write that case and confirm the grader catches it. A grader satisfied by silence — an empty result, a missing file, a command that errored before producing output — must be rewritten to require a positive signal.

### 4. Score honestly
Report pass@k for capability and pass^k for reliability — they answer different questions, and reliability is usually the one that matters in production. State the sample size. A 5-case suite that passes tells you very little; say so rather than reporting a percentage that implies precision you do not have.

## Rules

- **Never write "verified" or "tested" for a check that has not proven its own field of view.** That phrase converts a weak result into a strong claim, and it is the error this suite exists to prevent.
- Prefer many small deterministic assertions over one large model-graded judgement.
- Test the behaviour, not the wording. An eval that breaks when the phrasing changes measures the phrasing.
- **Do not add a verification step that re-checks work the system already verifies.** Redundant verification inflates cost and hides which layer actually caught the failure.
- When a case fails, report the failure with its actual output. Do not summarise it as "minor" — severity is the reader's call, not yours.
- Record what you could NOT test and why. An unstated blind spot reads as coverage.

## Output
A runnable suite (script or structured cases), a table of case → grader → pass criterion → failure mode, the seeded positive control with evidence it fired, and a short list of behaviours left untested with the reason.