robots.txt is a plain-text file at the root of your domain that tells crawlers, including the growing list of AI-specific ones, which parts of your site they may access. Each AI crawler identifies itself with its own user-agent string, which means you can allow or block them individually rather than as one undifferentiated group. This guide gives copy-paste rules for the major AI crawlers, explains the syntax that makes them work, and covers the mistakes that most commonly cause a rule to block more, or less, than intended.
One file, sitting quietly at yoursite.com/robots.txt, is the single most direct lever you have over which AI systems can read your site at all.
How robots.txt actually works
robots.txt follows the Robots Exclusion Protocol, a long-standing, voluntary convention where a crawler checks a specific file at the root of a domain before crawling, and respects the rules found there. It is voluntary in the sense that nothing technically forces a crawler to obey it, but every major, reputable crawler, including the AI crawlers covered here, states in its own documentation that it checks and respects robots.txt. The file is organized into blocks, each starting with a User-agent line naming which crawler the following rules apply to, followed by one or more Disallow or Allow lines specifying paths.
User-agent: GPTBot
Disallow: /private/
Allow: /
User-agent: *
Disallow:The first block targets one specific crawler by name. The second, using User-agent: *, applies to any crawler not matched by a more specific block above it. Rule specificity, not file order in a strict sense, generally determines which block a given crawler follows, so a crawler matching a named block uses that block's rules rather than falling through to the wildcard.
These decisions connect directly to your broader visibility goals: if you are actively pursuing citation from tools like Perplexity, reviewing how Perplexity picks sources alongside this guide clarifies which crawlers matter most for that specific product.
Copy-paste recipes for the major AI crawlers
Below are working rule blocks for the crawlers most relevant to AI visibility. Replace with your actual paths and combine as many blocks as apply to your situation.
# Allow OpenAI's training crawler
User-agent: GPTBot
Allow: /
# Allow OpenAI's live search/browsing crawler
User-agent: OAI-SearchBot
Allow: /
# Allow Anthropic's crawler
User-agent: ClaudeBot
Allow: /
# Allow Google's AI-training-specific crawler
User-agent: Google-Extended
Allow: /
# Allow Perplexity's crawler
User-agent: PerplexityBot
Allow: /
# Allow Common Crawl, used by many downstream AI systems
User-agent: CCBot
Allow: /Full context on what GPTBot specifically does and the tradeoffs of allowing or blocking it are covered in GPTBot explained, and every crawler in this list, along with several less common ones, is documented in more depth in the AI crawler directory.
If your goal is the opposite, blocking AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /Note the deliberate omission of OAI-SearchBot and PerplexityBot from a blanket block in many real-world configurations: site owners frequently want to prevent their content from training future models while still allowing it to be fetched and cited live during a user's search or browsing session, which is exactly the training-versus-retrieval distinction covered in GPTBot explained.
Table: matching your goal to a rule set
| Your goal | Rule pattern |
|---|---|
| Maximize AI visibility and citation | Allow all major AI crawlers explicitly |
| Block training inclusion, allow live citation | Disallow training-specific crawlers (GPTBot, Google-Extended, ClaudeBot), allow search/browsing crawlers (OAI-SearchBot) |
| Block all AI crawling entirely | Disallow every named AI crawler explicitly, do not rely on the wildcard alone |
| Selectively protect specific content only | Use a shared wildcard Allow at the root, then a specific Disallow path per protected section |
Named, explicit blocks per crawler are more reliable than relying on a single wildcard rule to cover every AI crawler at once.
Why crawl-delay and other legacy directives rarely matter here
Older robots.txt guides sometimes mention a Crawl-delay directive intended to throttle how fast a crawler requests pages. Most major AI crawlers do not document support for this directive, and it is not part of the more recent, formalized Robots Exclusion Protocol standard that many crawlers now reference. For AI-crawler management specifically, the two directives that matter in practice are Allow and Disallow under the correct User-agent block; spend your effort getting those right rather than adding legacy directives with inconsistent support.
A worked example: a personal site allowing citation but protecting one section
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow: /client-portal/
Disallow: /drafts/This configuration allows the named AI crawlers full access to the public site while the wildcard block, which applies to any crawler not explicitly named above it, keeps a private client portal and unpublished draft folder out of general crawling. Because the named blocks appear first and are specific, the AI crawlers listed follow their own permissive rules rather than inheriting the wildcard's restrictions, which only affects crawlers with no dedicated block of their own.
Sitemap declarations belong in the same file
robots.txt is also the conventional place to point crawlers toward your XML sitemap, which helps any crawler, AI-specific or otherwise, discover your full set of indexable pages efficiently rather than relying purely on following internal links:
Sitemap: https://yoursite.com/sitemap.xmlPlace this line anywhere in the file, commonly at the top or bottom. It applies globally regardless of which user-agent block it sits near, and including it costs nothing while meaningfully helping discovery for every well-behaved crawler that reads the file.
Common mistakes that break the intent of the file
- Relying only on the wildcard block. A rule under
User-agent: *does not automatically apply to a crawler matched by its own explicitly named block elsewhere in the file; named blocks take precedence for the crawler they name. - Blocking with a path typo.
Disallow: /Blogwill not match/blogon a case-sensitive server; paths must match exactly. - Forgetting the file is public. robots.txt is fully readable by anyone, so it is not a security or privacy mechanism, only a crawling-behavior instruction that well-behaved crawlers choose to respect.
- Assuming an old, inherited robots.txt reflects current intent. Many sites carry a blanket
Disallow: /from a staging environment that migrated to production without anyone revisiting the file. - Not testing after a site migration. A new CMS or hosting platform sometimes regenerates robots.txt automatically with different defaults than the previous one.
Visit yoursite.com/robots.txt directly in a browser. Read every block. For each AI crawler you recognize, confirm the rule matches what you actually intend, not what a default template happened to set years ago.
Keep the sitemap declaration current whenever your URL structure changes, since a stale sitemap URL pointing at a file that no longer exists, or that reflects an old site structure, undermines the discovery benefit it is meant to provide.
Combining robots.txt with an llms.txt file
robots.txt controls access; it does not describe your site's content or priorities to an AI system in any structured way. A newer, complementary convention, llms.txt, is meant to give AI systems a curated summary of a site's most important pages, covered in full in llms.txt for personal sites. The two files serve different purposes and are worth maintaining together: robots.txt says who may enter, llms.txt says where to look first once they are in.
How often to revisit these rules
New AI crawlers appear periodically as new products launch, and existing ones occasionally change their documented user-agent string. Treat robots.txt the way you would treat any piece of technical infrastructure with a slow but real rate of change: check it after any major site migration, check it whenever you hear about a new AI product relevant to your field, and otherwise review it roughly twice a year alongside other technical SEO housekeeping. A rule set that was comprehensive and correct eighteen months ago can quietly miss a crawler that did not exist when the file was last edited.
Keep a short changelog of your robots.txt edits, even just a comment line noting the date and reason for a change, since the file itself has no version history visible to a future editor and a one-line comment can save real confusion months later when someone asks why a particular crawler is blocked or allowed.
A comment line in robots.txt starts with a hash character and is ignored by crawlers entirely, so it is safe to use liberally for exactly this kind of internal documentation without affecting how any crawler parses the actual rules.
None of this changes how a crawler behaves, but it turns a plain rule file into something a future collaborator, or you yourself a year from now, can actually understand at a glance.
Validating your robots.txt file
After editing the file, fetch it directly at your domain's root and read it back carefully, since a syntax error, a stray character, or a misplaced colon can silently change which rules apply. Google's Search Console includes a robots.txt testing tool for checking how Googlebot specifically parses your file, and while equivalent public testing tools are not available for every AI crawler individually, the underlying syntax is standard enough that a rule verified as syntactically correct for one well-behaved crawler generally parses correctly for the others as well.
FAQ
How do I allow a specific AI crawler like GPTBot in robots.txt? +
Does a wildcard User-agent: * rule apply to all AI crawlers automatically? +
Can I block AI training crawlers while still allowing live citation in AI search tools? +
Is robots.txt a security mechanism for hiding private content? +
What is the difference between robots.txt and llms.txt? +
How do I check whether my robots.txt rules are working as intended? +
Find out what AI says about you today.
Start with a baseline. See the exact words the engines return about your name, then decide.
Claim your name →