# AI Systems Engineer — Job Description Template *A practical JD for the person who makes AI systems actually work in production — not a prompt hobbyist, not a research scientist. Genericized from how Engineering Square writes and vets the role.* Replace the bracketed placeholders, cut what doesn't apply, and keep the skills section honest: hire for the work that determines whether an AI system survives contact with production, not for the buzzwords on a résumé. --- ## The role in one paragraph An AI Systems Engineer builds and operates LLM-powered systems end to end: defining what "correct" means as a measurable eval, plumbing the data that feeds the model, designing the prompts and tools the model uses, and wiring the observability that tells you when it drifts. This is a software engineering role with a specialty, not a prompt-writing role. The output is a production system with a golden set, a cost/latency budget, and monitoring — not a clever demo. --- ## What you'll do - Turn fuzzy "make it use AI" requests into specified tasks with acceptance criteria and a golden eval set. - Build the data pipelines that feed models: retrieval (RAG) over governed corpora, chunking and indexing, freshness and access control. - Design prompts, tool/function schemas, and agent control flow — and the guardrails that keep them inside the data boundary. - Stand up evals as code: golden sets, scoring (exact-match, rubric, LLM-as-judge validated against humans), and CI that runs them on every change. - Instrument everything: token cost, p50/p95 latency, pass rates, refusal correctness, and drift — with alerts, not dashboards nobody reads. - Right-size the model. Test the cheaper option; ship the smallest model that clears the bar; keep the model swappable when the frontier moves. - Own the system in production: on-call for the AI surface, incident response, and turning every surprising failure into a new golden case. --- ## Skills that actually matter *Weighted for what breaks in production, not what's fashionable.* ### Must have - **Strong general software engineering.** Writes production code in at least one modern language, reads a codebase, ships tested increments. AI systems fail on plumbing far more than on model quality. - **Evals and measurement.** Can design a golden set, pick a scoring method, and defend a pass threshold with numbers. Treats "which model is better" as a table to sort, not a vibe. - **Data plumbing.** Retrieval pipelines, embeddings and vector search, chunking strategy, and — critically — data-access boundaries and PII handling. Knows what is allowed to enter a prompt and what never is. - **Prompt and tool design.** Structured outputs, function/tool schemas, and agent control flow. Understands context windows, token budgets, and why deterministic structure beats clever phrasing. - **Observability.** Logs, traces, cost and latency metrics, and alerting for a non-deterministic system. Can tell you what a request cost and why it was slow. - **Judgment about failure modes.** Prompt injection, hallucination, silent degradation, cost blowups. Designs for the wrong answer, not just the happy path. ### Nice to have - Experience with an agent framework (LangChain / LlamaIndex / CrewAI or equivalent) and the Model Context Protocol (MCP). - Fine-tuning or distillation experience, and the judgment to know when *not* to reach for it. - Self-hosted/open-weight model deployment for in-boundary inference. - Background in the domain the system serves (support, sales, healthcare, finance) — enough to know what a correct answer looks like. ### Explicitly not required - A PhD or published research. This is an engineering role, not a research role. - "10 years of LLM experience." The field is younger than that; anyone claiming it is rounding up. - Memorized benchmark leaderboards. We care that you can build a golden set for *our* task, not recite MMLU scores. --- ## What we don't want - A prompt tinkerer with no software engineering foundation. - Someone who reaches for the biggest frontier model by reflex and has never measured whether a smaller one would do. - Anyone who ships an AI feature with no eval set, no cost budget, and no monitoring, and calls the demo "done." --- ## Compensation - **Base salary band:** `$[LOWER] – $[UPPER] [CURRENCY]` — *state a real range; candidates screen out silently on "competitive."* - **Equity / bonus:** `[RANGE OR N/A]` - **Location / remote:** `[POLICY]` - **Level:** `[IC3 / Senior / Staff — map to your ladder]` --- ## Interview loop *Mirrors how we vet: a structured screen, then evidence they can actually build, then judgment under pressure. No brain-teasers, no take-homes that cost a week of unpaid work.* | Stage | Format | What it tests | |---|---|---| | **1. Structured screen** | 30–45 min, fixed questions, scored on a rubric | Real project ownership; can they explain a system they built and why it worked? | | **2. Live technical** | 60–90 min, pairing on a real problem | Can they write and debug production code, not just talk about it? | | **3. Eval design exercise** | 45–60 min, whiteboard/doc | Given a fuzzy AI task, can they define acceptance criteria, a golden set, and a scoring method? This is the load-bearing round. | | **4. Systems & failure-mode discussion** | 45 min | Cost/latency trade-offs, data boundaries, what breaks in production and how they'd catch it. | | **5. Values / collaboration** | 30 min | How they handle disagreement, review, and being wrong. | **Scoring discipline:** every interviewer scores against a written rubric before seeing anyone else's scores. Structured, comparable evidence — not a vibe check in the hallway afterward. --- ## How to use this template 1. **Cut ruthlessly.** A JD listing thirty "requirements" screens out the good candidates who are honest about not having all thirty. Keep the must-haves short and true. 2. **State the salary band.** It is the single biggest filter for wasted interview cycles on both sides. 3. **Keep the eval-design round.** It is the one stage that separates people who can *build* AI systems from people who can *talk about* them. If you cut one round, do not cut that one. > Hire the engineer who defines "correct" as a number and instruments the system to prove it — not the one with the most model names on their résumé. --- *Template by Engineering Square. Reuse freely inside your organization.*