The first question on most AI projects is 'which model should we use?' It's the wrong first question. Model choice is the last decision in our process, not the first, because it's the one decision that's cheap to change and easy to get right once everything upstream is settled. Swapping Claude for a GPT-5-class model or a Gemini variant is a config change and a re-run of your eval suite. What's expensive — and what actually determines whether the system works — is everything you have to decide before the model is even relevant: where your data lives and what's allowed to leave its boundary, what 'correct' means in numbers you can measure, and what latency and cost the use case can tolerate. Settle those, and the model practically selects itself.
We run this as two phases before any model is chosen. We call them Survey and Blueprint, and skipping them is the single most common reason AI pilots stall in the demo phase and never reach production.
Survey: data boundaries before anything
The Survey phase answers one question: what data does this system touch, and where is it allowed to go? Before a single prompt is written, we map the data the feature needs, classify it, and draw the boundary it can't cross. This is what determines whether you can call a hosted API at all, whether you need a zero-retention or in-region deployment, and whether some of the work has to run on infrastructure you control.
- What categories of data enter the prompt — public, internal, customer PII, regulated (health, financial), or secrets that must never reach a third party.
- Where inference is allowed to happen — a vendor API, a zero-retention enterprise tier, an in-region endpoint, or self-hosted on our own hardware.
- What's retrievable versus what's baked in — RAG over a governed corpus keeps sensitive data at arm's length and auditable, instead of hoping the model memorized the right thing.
- Who is accountable for a wrong or leaked answer — because that dictates how much of the pipeline has to be inspectable.
Get this wrong and no model is the right model — you'll have built a demo you're legally not allowed to ship. Get it right and the field of viable models is already narrowed to the ones that can operate inside your boundary, which is often a much shorter list than the leaderboard suggests.
Blueprint: evals and budgets before selection
The Blueprint phase turns 'it seems to work' into numbers. Two artifacts come out of it, and both must exist before we compare models:
- 01An eval set — a representative, versioned collection of real inputs with a defined notion of a correct or acceptable output, plus a way to score it (exact match, rubric-graded, an LLM judge you've spot-checked against humans, or task success for agents). Twenty to fifty real cases beats a thousand synthetic ones.
- 02A cost and latency budget — the ceiling per request in dollars and milliseconds that the use case can actually tolerate, derived from the product, not from the model's capabilities.
Without an eval set, 'which model is better' is a vibe. With one, it's a table you can sort — accuracy against cost against p95 latency — and the decision makes itself.
The budget is what keeps teams honest about right-sizing. A frontier model at the top of every benchmark is the obvious pick until you notice the task is a classification the size of a tweet, running ten thousand times an hour, where a smaller and cheaper model clears your eval bar at a fraction of the latency and cost. Frontier-by-default is a habit, not a strategy. The eval set tells you the smallest model that's good enough, and 'good enough, measured' is almost always the right production answer.
Why the ordering matters
When model choice comes last, every earlier decision constrains it in a useful way. The data boundary rules out models that can't run where they're allowed. The eval set ranks the survivors on the dimension you actually care about. The budget knocks out the ones that are accurate but too slow or too expensive for the workload. What's left is usually a small handful, and picking among them is a low-stakes call you can revisit any time the frontier moves — which it does, constantly.
This is also why our AI work survives model churn that strands other teams. A new release lands, we point the eval suite at it, and either it beats the incumbent on our numbers and we switch, or it doesn't and we don't. There's no rearchitecture, no scramble, no debate about vibes — because the model was always the swappable part. The boundary, the evals, and the budget are the system. The model is just the component you can replace without anyone noticing.