# LLM Acceptance-Criteria & Eval Worksheet

*Fill this out before you choose a model. If you cannot complete it, you are not ready to build — you are ready to prototype, which is a different, cheaper thing.*

Engineering Square runs this in the **Blueprint** phase of every AI engagement, after the **Survey** phase has settled where data is allowed to go. The premise: picking the model is the *last* decision, not the first. Everything on this worksheet constrains the model choice, so that by the end the model practically selects itself — and stays swappable when the frontier moves.

Copy this file, delete the guidance italics, and fill in every field.

---

## 1. Task definition

*One task per worksheet. If you have three tasks, you have three worksheets.*

| Field | Your answer |
|---|---|
| **Task name** | |
| **One-sentence description** | *What goes in, what comes out. A stranger should understand it.* |
| **Input** | *Exact shape: fields, formats, typical and max size.* |
| **Output** | *Exact shape: structured (JSON schema?) or free text? Length bound?* |
| **Who/what consumes the output** | *A human reads it? Another system parses it? An agent acts on it?* |
| **Blast radius of a wrong answer** | *Cosmetic / annoying / expensive / unsafe. This sets how hard the review gates must be.* |
| **Volume** | *Requests per hour at launch and at 12-month projection.* |

**Data boundary (carried from the Survey phase — restate it here):**

- Data categories entering the prompt: `public / internal / customer PII / regulated / secrets`
- Where inference is allowed: `vendor API / zero-retention tier / in-region / self-hosted`
- Anything that must **never** leave your infrastructure: ______________________

> If the data boundary rules out hosted APIs, note it now. No model is the right model if you are not allowed to send it the data.

---

## 2. Golden set

*The single highest-leverage artifact in the whole process. Without it, "which model is better" is a vibe. With it, it's a table you can sort.*

| Field | Your answer |
|---|---|
| **Golden set size** | *20–50 real cases beats 1,000 synthetic ones. Start at 20 real, grow with production failures.* |
| **Source of cases** | *Real historical inputs > handwritten. Where are they coming from?* |
| **Coverage** | *List the categories/edge cases the set must include (happy path, empty, adversarial, long input, ambiguous, multilingual…).* |
| **Where it's versioned** | *The golden set is code. It lives in the repo, reviewed like code.* |
| **Refresh cadence** | *Every production failure that surprises you becomes a new golden case.* |

**Coverage checklist — one row per category, aim for a real example of each:**

- [ ] Typical / happy path
- [ ] Empty or missing input
- [ ] Maximum-size input
- [ ] Ambiguous input (reasonable people disagree on the answer)
- [ ] Adversarial / prompt-injection attempt
- [ ] Out-of-scope input (the model should decline or defer)
- [ ] Known past failure (a real case that burned you before)

---

## 3. Scoring method

*How a single output gets turned into a pass/fail or a score. Pick the cheapest method that actually discriminates.*

| Method | Use when | Your choice |
|---|---|---|
| **Exact / structured match** | Output is deterministic (classification, extraction to a schema) | ☐ |
| **Rubric-graded (human)** | Quality is subjective but you can write a rubric | ☐ |
| **LLM-as-judge** | Volume is too high for humans; **you have spot-checked the judge against human labels** | ☐ |
| **Task success (agents)** | The output is an action; did the end state come out right? | ☐ |

**If using an LLM judge:** how many of its verdicts did you hand-check against a human, and what was the agreement rate? `____ / ____ checked, ____% agreement.` An unvalidated judge is just a second vibe.

---

## 4. Pass thresholds

*Numbers, not adjectives. "Good enough" must be a percentage before you start comparing models.*

| Metric | Threshold to ship | Rationale |
|---|---|---|
| **Overall pass rate on golden set** | `≥ ____%` | |
| **Pass rate on the must-never-fail subset** | `= 100%` on `____` cases | *The safety-critical cases have their own bar.* |
| **Max acceptable regression vs current** | `≤ ____%` | *For model swaps: how much can any single category drop?* |
| **Refusal / abstention correctness** | `≥ ____%` | *Does it decline when it should, and only then?* |

---

## 5. Cost & latency budget

*Derived from the product, not from the model's capabilities. This is what keeps you honest about right-sizing instead of reaching for the frontier by reflex.*

| Metric | Budget | Notes |
|---|---|---|
| **Cost per request** | `≤ $____` | *Multiply by projected volume — does the unit economics survive?* |
| **Cost per 1,000 requests** | `≤ $____` | |
| **p50 latency** | `≤ ____ ms` | |
| **p95 latency** | `≤ ____ ms` | *The tail is what users feel. Budget it explicitly.* |
| **Max tokens in / out** | `____ / ____` | *Context and output caps that keep cost bounded.* |
| **Throughput needed** | `____ req/s peak` | *Rate limits and concurrency on the chosen provider.* |

> A frontier model at the top of every benchmark is the obvious pick until you notice the task is a tweet-sized classification running ten thousand times an hour, where a smaller model clears your bar at a fraction of the cost and latency. Let the budget knock those out.

---

## 6. Human-review gates

*Where a human stays in the loop, and what triggers escalation. Set by the blast radius from Section 1.*

| Gate | Trigger | Who reviews | SLA |
|---|---|---|---|
| **Pre-release** | Every model/prompt change runs the full golden set | Eng owner | Before merge |
| **Sampled production review** | `____%` of live outputs sampled | | Weekly |
| **Confidence / uncertainty escalation** | Model signals low confidence or hits a guardrail | | Real-time |
| **User-flagged** | End user reports a bad answer | | `____ hrs` |

- [ ] Every mutating action an agent can take has a defined human-approval or reversibility path.
- [ ] There is a kill switch to disable the feature without a redeploy.

---

## 7. Model comparison table

*The payoff. Fill one row per candidate. When the boundary, evals, and budget are set, the winner is usually obvious — and picking among the survivors is a low-stakes, revisit-anytime call.*

| Model | Golden-set pass rate | Cost / 1k | p95 latency | Fits data boundary? | Notes |
|---|---|---|---|---|---|
| *Candidate A* | | | | ☐ | |
| *Candidate B* | | | | ☐ | |
| *Candidate C (smaller/cheaper)* | | | | ☐ | *Always test at least one you expect to be "too small" — it often clears the bar.* |

**Decision:** ship `____________`, because it is the smallest/cheapest model that clears every threshold inside the data boundary. Revisit when the frontier moves — point the golden set at the new release; switch only if it beats the incumbent on these numbers.

---

## 8. Weekly performance report

*Once in production, the system reports on itself. Keep this table live.*

| Week | Golden-set pass rate | Sampled prod accuracy | Cost / 1k | p95 latency | Incidents | New golden cases added |
|---|---|---|---|---|---|---|
| | | | | | | |
| | | | | | | |

**Standing questions each week:**

- Did the pass rate move? Why?
- Did any never-fail case fail? (If yes, that is an incident, not a metric.)
- Did cost or latency drift past budget?
- What production failure this week became a new golden case?

---

## How to use this well

1. **Fill Sections 1–5 before writing a single prompt.** The prompt is an implementation detail of a task you have already specified.
2. **Grow the golden set from real failures.** Every surprise in production is a missing test. Add it, and it can never surprise you the same way twice.
3. **Treat the model as the swappable component.** The boundary, the evals, and the budget are the system. When a new model lands, you re-run a table — you do not rearchitect.

> Without a golden set, "which model is better" is an opinion. With one, it's a table you sort by accuracy, cost, and p95 latency — and the decision makes itself.

---

*Worksheet by Engineering Square. Reuse freely inside your organization.*