How to Allow AI Crawlers in robots.txt (GEO Strategy)
Explicitly allow GPTBot, ClaudeBot, and PerplexityBot in robots.txt to get your content cited in AI search answers.
robots.txt Syntax — Allow AI Crawlers (Allow)
COPY-PASTE READY
User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: /
What This Does
Allowing AI crawlers explicitly means your content is eligible to be cited in AI search answers (GEO — Generative Engine Optimisation). Sites that are blocked from AI crawlers are excluded from ChatGPT, Claude, and Perplexity's knowledge base.
Validate your robots.txt live — check AI bot coverage and get a corrected file.
Open robots.txt Validator →