Engineering Square runs this in the Blueprint phase of every AI engagement, after the Survey phase has settled where data is allowed to go. The premise: picking the model is the last decision, not the first. Everything on this worksheet constrains the model choice, so that by the end the model practically selects itself — and stays swappable when the frontier moves. If you cannot complete it, you are not ready to build; you are ready to prototype, which is a different, cheaper thing.
1. Task definition
One task per worksheet. If you have three tasks, you have three worksheets.
| Field | What to fill in |
|---|---|
| Task name | A short handle for this one task. |
| One-sentence description | What goes in, what comes out. A stranger should understand it. |
| Input | Exact shape: fields, formats, typical and max size. |
| Output | Exact shape: structured (JSON schema?) or free text? Length bound? |
| Consumer of the output | A human reads it? Another system parses it? An agent acts on it? |
| Blast radius of a wrong answer | Cosmetic / annoying / expensive / unsafe — this sets how hard the review gates must be. |
| Volume | Requests per hour at launch and at 12-month projection. |
Then restate the data boundary carried from the Survey phase: which data categories enter the prompt (public, internal, customer PII, regulated, secrets), where inference is allowed to happen (vendor API, zero-retention tier, in-region, self-hosted), and anything that must never leave your infrastructure. If the boundary rules out hosted APIs, note it now — no model is the right model if you are not allowed to send it the data.
2. Golden set
The single highest-leverage artifact in the whole process. Without it, 'which model is better' is a vibe; with it, it's a table you can sort. Record the set size (20–50 real cases beats 1,000 synthetic ones), the source of the cases (real historical inputs beat handwritten), the coverage it must include, where it's versioned (the golden set is code — it lives in the repo, reviewed like code), and the refresh cadence.
Coverage checklist — aim for a real example of each:
- Typical / happy path
- Empty or missing input
- Maximum-size input
- Ambiguous input (reasonable people disagree on the answer)
- Adversarial / prompt-injection attempt
- Out-of-scope input (the model should decline or defer)
- Known past failure (a real case that burned you before)
3. Scoring method
How a single output gets turned into a pass/fail or a score. Pick the cheapest method that actually discriminates.
| Method | Use when |
|---|---|
| Exact / structured match | Output is deterministic (classification, extraction to a schema). |
| Rubric-graded (human) | Quality is subjective but you can write a rubric. |
| LLM-as-judge | Volume is too high for humans, AND you have spot-checked the judge against human labels. |
| Task success (agents) | The output is an action; did the end state come out right? |
If you use an LLM judge, record how many of its verdicts you hand-checked against a human and the agreement rate. An unvalidated judge is just a second vibe.
4. Pass thresholds
Numbers, not adjectives. 'Good enough' must be a percentage before you start comparing models.
| Metric | Set a threshold |
|---|---|
| Overall pass rate on golden set | ≥ your number % |
| Pass rate on the must-never-fail subset | = 100% on the named safety-critical cases |
| Max acceptable regression vs current | ≤ your number % (how much any single category can drop on a swap) |
| Refusal / abstention correctness | ≥ your number % (declines when it should, and only then) |
5. Cost & latency budget
Derived from the product, not from the model's capabilities. This is what keeps you honest about right-sizing instead of reaching for the frontier by reflex.
| Metric | Budget |
|---|---|
| Cost per request | ≤ $ — multiply by projected volume; does the unit economics survive? |
| Cost per 1,000 requests | ≤ $ |
| p50 latency | ≤ ms |
| p95 latency | ≤ ms — the tail is what users feel; budget it explicitly |
| Max tokens in / out | context and output caps that keep cost bounded |
| Throughput needed | req/s peak — rate limits and concurrency on the chosen provider |
A frontier model at the top of every benchmark is the obvious pick until you notice the task is a tweet-sized classification running ten thousand times an hour, where a smaller model clears your bar at a fraction of the cost. Let the budget knock those out.
6. Human-review gates
Where a human stays in the loop, and what triggers escalation — set by the blast radius from Section 1.
| Gate | Trigger | SLA |
|---|---|---|
| Pre-release | Every model/prompt change runs the full golden set | Before merge |
| Sampled production review | A set percentage of live outputs sampled | Weekly |
| Confidence / uncertainty escalation | Model signals low confidence or hits a guardrail | Real-time |
| User-flagged | End user reports a bad answer | Your hours |
Confirm two things before launch: every mutating action an agent can take has a defined human-approval or reversibility path, and there is a kill switch to disable the feature without a redeploy.
7. Model comparison table
The payoff. Fill one row per candidate. When the boundary, evals, and budget are set, the winner is usually obvious — and picking among the survivors is a low-stakes, revisit-anytime call. Always test at least one model you expect to be 'too small'; it often clears the bar.
| Model | Pass rate | Cost / 1k | p95 latency | Fits boundary? |
|---|---|---|---|---|
| Candidate A | — | — | — | yes / no |
| Candidate B | — | — | — | yes / no |
| Candidate C (smaller/cheaper) | — | — | — | yes / no |
Decision: ship the smallest/cheapest model that clears every threshold inside the data boundary. Revisit when the frontier moves — point the golden set at the new release and switch only if it beats the incumbent on these numbers.
8. Weekly performance report
Once in production, the system reports on itself. Keep this table live.
| Week | Golden-set pass | Prod accuracy | Cost / 1k | p95 | Incidents | New cases |
|---|---|---|---|---|---|---|
| — | — | — | — | — | — | — |
Standing questions each week: did the pass rate move and why; did any never-fail case fail (if yes, that's an incident, not a metric); did cost or latency drift past budget; and what production failure this week became a new golden case?
How to use this well
- 01Fill Sections 1–5 before writing a single prompt. The prompt is an implementation detail of a task you have already specified.
- 02Grow the golden set from real failures. Every surprise in production is a missing test; add it, and it can never surprise you the same way twice.
- 03Treat the model as the swappable component. The boundary, the evals, and the budget are the system; when a new model lands, you re-run a table, you do not rearchitect.
Without a golden set, 'which model is better' is an opinion. With one, it's a table you sort by accuracy, cost, and p95 latency — and the decision makes itself.