AI Crawlers and robots.txt: The Full Directory

TL;DR

Every major AI lab runs multiple crawlers with different jobs: one feeds model training, one builds a search index, one fetches pages live when a user asks. Your robots.txt controls each one by name, and blocking the wrong one makes you invisible at the exact moment a buyer asks an engine about you. Here is the full directory, the exact user agents, and two copy-paste robots.txt configurations for a personal site.

Most robots.txt advice is written for publishers worried about content theft. If your name is the product, the calculation inverts: the crawler is not stealing your work, it is delivering your reputation. You still need to know exactly who is knocking.

Why does one AI company send three different crawlers?

Because an AI answer is assembled from three different supply chains, and each has its own bot. OpenAI is the cleanest example, documented at platform.openai.com/docs/bots:

GPTBot collects public web content that may be used to train future models. This is the long game: what GPTBot reads today can shape what a model "knows" from memory next year.
OAI-SearchBot builds and refreshes the index behind ChatGPT search. This is the medium game: it determines whether your pages are candidates for citation when ChatGPT searches the web.
ChatGPT-User fetches a page on demand when a user's question sends ChatGPT to your site right now. This is the short game: the live visit during a conversation that mentions you.

Each responds to its own name in robots.txt, and blocking one does not block the others. That separation is not bureaucracy; it is a control panel. It lets you say "do not train on me, but do cite me," or the reverse. Most site owners never touch the panel and get the default: whatever their CMS or CDN decided for them. If you want the mechanics of what each pipeline does with your name once the crawler leaves, that is the subject of training data vs retrieval.

The AI crawler directory

The named crawlers that matter for a personal site in 2026, with what blocking each one actually costs you:

User agent	Operator	Job	If you block it
`GPTBot`	OpenAI	Training data collection	Future OpenAI models stop learning from your site
`OAI-SearchBot`	OpenAI	ChatGPT search index	Your pages stop being citable in ChatGPT search
`ChatGPT-User`	OpenAI	Live fetch during user chats	ChatGPT cannot open your site when asked about you
`ClaudeBot`	Anthropic	Training data collection	Future Claude models stop learning from your site
`Claude-SearchBot`	Anthropic	Claude search indexing	Your pages fall out of Claude's search results
`Claude-User`	Anthropic	Live fetch during user chats	Claude cannot open your site on request
`PerplexityBot`	Perplexity	Index behind cited answers	You stop appearing as a Perplexity source
`Perplexity-User`	Perplexity	Live fetch for user actions	Perplexity cannot visit your pages mid-answer
`Google-Extended`	Google	Control token for Gemini training	Gemini training loses your content; normal Google Search is unaffected
`Googlebot`	Google	Classic search crawling	You vanish from Google, including the search that feeds AI features
`CCBot`	Common Crawl	Open web archive used in many training sets	You exit a dataset many labs train on

Three sources anchor this table and are worth bookmarking: OpenAI's bot documentation, Anthropic's crawler help page covering ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity's crawler documentation, which also publishes the IP ranges each of its agents uses. Note the odd one out: Google-Extended is not a crawler at all. Googlebot does the fetching, and Google-Extended is a robots.txt token that tells Google whether the fetched content may feed Gemini training.

What should a personal site put in robots.txt?

Two honest configurations cover almost every individual. Pick by intent, not by fashion.

Configuration one: fully open

For anyone whose goal is to be known, recommended and named by machines. This is the PEO default:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

No AI-specific rules at all. Every declared crawler, training or retrieval, reads everything. Your only jobs are to keep the sitemap current and make sure nothing upstream, a CDN default, a WordPress plugin, a bot-protection rule, is silently blocking crawlers your robots.txt welcomes. Check your server logs or your CDN's bot dashboard for the user agents above; absence of visits is a finding, not a comfort.

Configuration two: cite me, do not train on me

For those uneasy about model training but unwilling to disappear from AI answers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Understand the trade before you ship it. Blocking training bots means future models are less likely to know you from parametric memory, so your entire AI presence rides on retrieval: being found and cited at question time. That is a legitimate strategy, but it is a bet on one door instead of two.

The asymmetry

For a publisher, an AI crawler is a cost: content out, nothing back. For an individual, it is distribution: the crawler is how engines learn there is a you to recommend. AI referral traffic is small, around 1 percent of the total across industries, but ChatGPT referrals convert at 14 to 16 percent against roughly 1.76 percent for Google organic, per Conductor's 2026 benchmarks via SEO Sherpa's statistics roundup. Blocking the pipes that carry that traffic is a strange way to grow a name.

What about the crawlers this table leaves out?

The directory above covers the agents with direct consequences for whether AI engines can name you. A second tier exists and deserves a sentence each. Meta publishes crawlers for its AI training, controllable by user agent like the rest. Apple offers Applebot-Extended, which works like Google-Extended: not a separate crawler but a token telling Applebot whether fetched content may feed Apple's AI training. And there is a long tail of aggressive scrapers, ByteDance's Bytespider being the most frequently named, that have been widely reported to ignore robots.txt entirely, which is a useful reminder of what the file is and is not. For an individual running a PEO strategy none of these change the decision much: the engines your buyers actually ask are the ones in the table.

The other omission is deliberate: this directory will age. Labs rename agents, split them, and launch new ones without ceremony, which is why the configurations above control bots by name but default open, and why a twice-yearly review of the file belongs on your calendar in a way it never did before 2024.

Do AI crawlers actually obey robots.txt?

The declared ones do, on the record. OpenAI, Anthropic and Perplexity all document robots.txt compliance for their named agents, and Anthropic states its bots also respect the crawl-delay extension. But keep two caveats in view. First, robots.txt is a convention with no enforcement layer; a scraper that ignores it faces no technical barrier, and undeclared crawling remains a live controversy in the industry. Second, compliance is per-agent: a rule for GPTBot says nothing to OAI-SearchBot. The practical routine is to verify rather than trust: both OpenAI and Perplexity publish the IP ranges their bots crawl from, so a suspicious visitor claiming to be PerplexityBot can be checked against Perplexity's published JSON before you conclude anything.

How do I see which AI crawlers actually visit my site?

Stop guessing and read the guest book. Three ways in, from easiest to most precise:

Your CDN or host's bot analytics. Cloudflare, Vercel, Netlify and most managed WordPress hosts now break out AI crawler traffic by name in their dashboards. Five minutes, no setup, and you immediately know whether GPTBot has ever seen your about page.
Raw access logs. Filter for the user agent strings in the table above. Look at which paths each bot requests and how often. A retrieval bot hitting your homepage weekly but never reaching your best article is a routing problem you can fix with internal links and a current sitemap.
A canary page. Publish a new page, submit the updated sitemap, and time how long each crawler takes to fetch it. That interval is your personal answer to "how fast does new material about me become citable," and it varies enough between sites to be worth measuring rather than assuming.

Whatever you find becomes the baseline for the framework below. Crawl data tells you whether machines can read you; only visibility testing tells you what they say. You need both numbers.

Verifying a crawler is who it claims to be

Anyone can set a User-Agent header to say GPTBot or ClaudeBot, so a name in your access log is a claim, not a proof. The reputable labs solve this by publishing verifiable IP ranges alongside their user agent strings, and Perplexity in particular distributes its crawler IP ranges as a JSON file so a site operator can cross-check a visitor's address before trusting the label. The more portable technique, usable for any declared crawler whether or not it publishes ranges, is a double DNS lookup: resolve the visiting IP address back to a hostname, confirm that hostname belongs to the operator's domain, then resolve that hostname forward again and confirm it matches the original IP. That pattern is the same one long recommended for verifying Googlebot, and it works just as well for GPTBot or ClaudeBot. Skipping verification costs little for casual curiosity, but it matters a great deal before you change monitoring or firewall rules based on who you believe is visiting.

llms.txt: the crawler directory's companion file

robots.txt tells a crawler where it may go. It says nothing about what your site actually is, and that gap is what llms.txt was proposed to fill: a plain-language file at the root of your domain that summarizes who you are, what your site covers, and which pages matter most, written for a language model rather than a search index. Where robots.txt is a set of access rules, llms.txt is closer to a curated table of contents. Pairing the two makes sense: robots.txt governs entry, llms.txt governs orientation once entry is granted. The full spec and a copy-paste template live in llms.txt for personal sites, and it is worth publishing alongside whichever robots.txt configuration you choose from this page, since one file grants access and the other tells the visitor what is worth reading first.

A worked example: robots.txt for a solo consultant's site

Imagine a consultant, call him Sam Okafor, launching a personal site to support a PEO strategy. He wants to be citable everywhere and has nothing to protect from training. Here is the reasoning he walks through before publishing his file.

Step ladder

1. Sam checks whether any content on his site is paywalled or licensed. It is not, so there is no reason to fence off a path.
2. He decides his default posture: fully open, matching configuration one above, because visibility is the entire point of the site.
3. He publishes a current sitemap and references it in robots.txt so every crawler, AI or otherwise, can find new pages quickly.
4. He writes an llms.txt file summarizing his expertise and linking his three most important pages, so a model that fetches his site knows where to look first.
5. He checks his host's bot dashboard a week later to confirm GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot have all actually visited, not just that the file permits them to.
6. He puts a reminder on his calendar for six months out to re-check the directory, since new agents appear without warning.

A six-step decision walk-through for a personal site with nothing to protect and everything to gain from visibility.

Sam's decision is the common case. The rarer case, someone actively protecting licensed material while still wanting to be found for their expertise, needs configuration two from earlier in this piece, applied only to the specific paths worth protecting rather than the whole domain. Either way, the file works alongside the identity signals covered in structuring your identity for machines, since an open door is only useful if what sits behind it is legible once a crawler walks through.

A decision framework in four questions

Is your name the product? If clients, employers or investors might ask an engine about you, run configuration one. Visibility is the whole point, and the mechanics of getting recommended by ChatGPT all assume the door is open.
Do you sell the content itself? Paywalled courses, licensed research, book manuscripts: block training bots on those paths specifically, keep your about page and public writing open. robots.txt rules take a path, not just a domain.
Are you managing a live reputation problem? Blocking crawlers does not remove what models already learned; it only stops corrections from getting in. Sites with something to fix should open the doors wider, then fix the source material.
Not sure? Default open, then measure. Watch your logs for AI user agents, and track your AI visibility monthly so the decision runs on data instead of vibes.

Review the file twice a year. New agents keep appearing, and the guide Search Engine Land maintains on AI crawlers is a reasonable changelog for the category. Fifteen lines of plain text decide whether machines can read you at all; audits of that file, and everything downstream of it, are part of our PEO service.

FAQ

Does blocking GPTBot remove me from ChatGPT? +

Not immediately, and not completely. GPTBot only gathers training data, so blocking it stops future model versions from learning from your site, but it does not touch ChatGPT search, which uses OAI-SearchBot, or live browsing, which uses ChatGPT-User. Whatever earlier models already learned also stays learned.

Do AI crawlers respect robots.txt? +

The major declared crawlers from OpenAI, Anthropic, Google and Perplexity document robots.txt compliance for their named user agents. But robots.txt is a convention, not an enforcement mechanism, and undeclared or misbehaving scrapers exist. Treat it as a signpost the reputable bots follow.

Should a personal website block AI crawlers? +

For most individuals, no. Your goal is to be found, cited and recommended, and blocking search or user-fetch bots makes you invisible at the exact moment a buyer asks about you. The defensible middle path is blocking training bots while allowing search and retrieval bots, but understand that this trades long-term model memory for short-term citations.

What is llms.txt and how does it relate to robots.txt? +

llms.txt is a companion file, not a replacement. robots.txt controls whether a crawler may access your pages at all. llms.txt, read once access is granted, orients a language model to what your site covers and which pages matter most. Publish both together for a complete signal.

How do I verify a crawler claiming to be GPTBot or ClaudeBot is real? +

Cross-check the visiting IP address against the operator's published ranges where available, or perform a reverse DNS lookup on the IP followed by a forward lookup to confirm the hostname resolves back to the same address and belongs to the operator's domain. A user agent string alone is only a claim.

Does llms.txt replace the need for schema.org markup? +

No. llms.txt is a plain-language orientation file read at the whole-site level. Structured data such as Person schema and sameAs give machines an unambiguous, page-level record of who you are. Use both together rather than one instead of the other.

Find out what AI says about you today.

Start with a baseline. See the exact words the engines return about your name, then decide.

Claim your name →