The traffic that used to come from a blue link now increasingly arrives as a citation inside an AI answer, and optimizing for it is a different job than classic SEO. Answer-engine optimization — getting ChatGPT, Claude, Perplexity, and Google's AI surfaces to quote your page — comes down to three things you can actually control: let the retrieval crawlers in, hand them machine-readable structure, and write the first paragraph so it answers the question outright. Miss any one and you're invisible to the engine even when you'd rank fine in a traditional index. The mechanics aren't mysterious, but they're specific, and most sites get at least one of the three wrong — usually by blocking the crawler that fetches pages at answer time, which no amount of good content can fix.
Here's the checklist we run on every page that's meant to be found, and why each item is load-bearing rather than nice-to-have.
First, let the right crawlers in
There are two distinct kinds of AI bot and they are not the same decision. Training crawlers (GPTBot, ClaudeBot, Google-Extended) fetch pages to improve future models. Retrieval crawlers (OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User) fetch a page live to answer a question a user is asking right now — and those are the ones that produce a citation with a link back to you.
- Always allow the retrieval bots. Blocking them is the single most common self-inflicted wound — the page never enters the answer because the engine literally can't read it at query time.
- Allowing training bots is a values call, not a traffic one; blocking them costs you no citations today. We allow them, but there's a defensible case either way.
- Never ship a blanket Disallow: / — and watch for a staging robots.txt leaking to production, which silently kills every crawler and no analytics tool will page you about it.
- Blocking bandwidth-only crawlers like Bytespider or CCBot is fine if you want to; they don't drive answer citations either way.
Then, hand them structure
AI crawlers do not execute JavaScript. A page whose content is rendered client-side is, to an answer engine, a blank page — so every word that matters ships server-rendered or static, never assembled in the browser. On top of that, JSON-LD tells the engine what the page is without it having to infer:
- 01Organization schema site-wide, Product with Offer and AggregateRating on commerce pages, and BlogPosting with author, publisher, datePublished, and dateModified on articles.
- 02FAQPage schema wherever there are real questions — Google dropped FAQ rich results in 2026, but the answer engines still parse the markup, so it keeps earning citations even though it no longer earns a snippet.
- 03Article, ItemList, and FAQPage present together correlate with markedly more AI citations than prose alone — the combination reads as a well-structured source the engine can lift from cleanly.
If the content only exists after JavaScript runs, it doesn't exist to the crawler. Server-render everything you want quoted — the answer engine reads the HTML, not the rendered page.
Then, write for extraction
Structure gets you read; the writing gets you quoted. Engines lift passages that answer a question in a self-contained way, so we front-load the answer and make the facts easy to excerpt:
- The first 150–200 words answer the primary query directly — no throat-clearing preamble the engine has to skip past to reach the substance.
- Tables, lists, and FAQ blocks over long prose; a bulleted claim is easier to lift verbatim than the same fact buried mid-paragraph.
- Two or three specific numbers per section — versions, dates, thresholds — because concrete figures are what an engine cites, and they signal the page is a primary source rather than a summary of one.
- Refresh the key pages every 60–90 days and bump dateModified; freshness is a ranking and citation signal, and a page dated last year loses to an equivalent one dated last month.
What doesn't move the needle
One thing to not waste effort on: as of mid-2026, no AI search platform reads /llms.txt. It's genuinely useful for IDE and coding agents on a docs or developer-tool site, so we ship it there — but on a marketing site it's decorative, and treating it as GEO leverage is a way to feel productive while the crawler you actually needed to allow is still blocked in robots.txt. The order of operations is the whole game: crawler access first, server-rendered structure second, extraction-ready copy third. Get those three right and citations follow; add llms.txt and skip the three, and nothing does.