How AI Engines Select and Cite Sources

AI engines use Retrieval-Augmented Generation (RAG): they query a vector index of crawled pages, retrieve the top candidates, then score each by authority, freshness, and answer quality. Pages with structured content, schema markup, and direct answers score higher and get cited.

Understanding this pipeline is essential to knowing which GEO optimizations matter most — and why.

The RAG Pipeline

User query
     ↓
Document retrieval (vector index search)
     ↓
Scoring: relevance + authority + freshness + structural quality
     ↓
Selection of candidate sources  ← this is where GEO impacts results
     ↓
Synthesis and generated answer with citations

Each step in this pipeline is an opportunity for optimization.

Step 1: Crawling and Indexing

Before an AI engine can cite your page, it must crawl and index it. Each major AI engine runs its own crawler:

Crawler	Engine	robots.txt User-agent
GPTBot	ChatGPT Search	GPTBot
OAI-SearchBot	OpenAI search	OAI-SearchBot
ClaudeBot	Anthropic training	ClaudeBot
Claude-User	Anthropic retrieval	Claude-User
Claude-SearchBot	Anthropic search	Claude-SearchBot
PerplexityBot	Perplexity	PerplexityBot
Google-Extended	Gemini / AI Overviews	Google-Extended

A critical problem: many sites have Disallow: / in robots.txt with wildcard rules that have no exceptions for AI crawlers. These sites are completely invisible to all generative systems.

Step 2: Vector Indexing

Crawled pages are processed into vector embeddings — numerical representations of semantic meaning. When a user asks a question, the AI engine searches this vector index for pages whose semantic content most closely matches the query.

This is why keyword stuffing doesn’t work for GEO. What matters is semantic clarity: does your page clearly answer the question? Headers framed as direct questions, answer capsules, and structured content improve semantic matching.

Step 3: Candidate Scoring

The retrieved candidates are scored across multiple dimensions before citation selection:

Relevance — How closely does the page answer the query? Pages that answer the question in the first 1-2 sentences of each section score higher.

Authority — Domain authority (backlinks, brand mentions, E-E-A-T signals). Brand mentions without links count 3:1 over backlinks for AI Overviews presence.

Freshness — When was the page published and last updated? The article:published_time and article:modified_time meta tags are the primary recency signals. Pages with recent lastmod in sitemaps also score better.

Structural quality — Does the page use schema markup? Is it semantically structured (article, section, time, cite elements)? Schema markup increases precise information extraction from 16% to 54% (Semrush, 10,000-page study).

Answer density — Does the page contain direct, self-contained answers? Answer capsules (40-60 words) are the most citeable content unit because they can be directly quoted without requiring surrounding context.

Step 4: Citation Selection

After scoring, the AI engine selects a small set of sources to cite in its generated answer — typically 3-10 pages. The selection favors:

1. Pages with the highest combined score (relevance + authority + freshness + structure)
2. Diverse source types (to avoid citing the same domain multiple times)
3. Pages that contributed unique factual claims to the synthesized answer

What This Means for Your Content

The RAG pipeline has clear implications for GEO implementation:

For crawling: robots.txt must explicitly allow each AI crawler by name. A general Allow: * rule may not be sufficient — use explicit User-agent entries.

For indexing: Content must be server-rendered HTML. Client-side rendered JavaScript (CSR) pages may not be crawled or indexed by AI systems that don’t execute JavaScript.

For scoring: Every content page needs article:published_time, JSON-LD Article schema, and inverted pyramid structure. These directly affect scoring factors.

For citation: Answer capsules at the start of each major section maximize the chance that your page contributes quotable content to the synthesized answer.

The Difference Between Google Search and AI Citation

Traditional Google search shows a ranked list of links. Users click through to your page. The metric is clicks.

AI citation is different: the engine synthesizes an answer and shows a citation alongside it. Users may read the AI answer without clicking through. The metric is brand mentions and citation visibility — not click-through rate.

This changes the optimization goal: instead of optimizing for click-through, you optimize for being cited. That means your content needs to be quotable, authoritative, and directly answer the question — not just rank for it.

Key Takeaways

• AI engines score pages on relevance, authority, freshness, and structural quality
• Schema markup and inverted pyramid structure are the highest-impact GEO optimizations
• robots.txt must explicitly list AI crawlers or they may be blocked
• Content must be server-rendered for AI crawler access
• Answer capsules (40-60 words, self-contained) are the most citeable content unit