robots.txt Builder for AI Crawlers — Set Your AI Training Preferences

Every website can tell crawlers what they may take. The file that does it is called robots.txt, and until now, telling AI companies “do not train on my site” meant a separate rule for every company. On October 30, 2026, the four big labs agreed to honor one shared signal for training — this builder writes it for you, together with the per-crawler rules that work today. Free, and it runs entirely in your browser: nothing you type is uploaded.

Not sure what the choices mean? There are three different kinds of AI visitor, and blocking each costs something different. Training crawlers collect pages to train models. Search and answer crawlers index pages so AI search can cite them — blocking these can cost you visibility and referrals. User-triggered fetchers visit a page because a person asked an assistant to — blocking them means assistants cannot read your pages for their users.

1. Start from a preset

2. Or set each choice yourself

The shared training signal

__usage_ai-training — the new shared opt-out token. One group that OpenAI, Google, Microsoft, and Anthropic have all said they will honor (on different dates — see the timeline below). It speaks about training only, not crawling.

Training crawlers — block these from taking my pages for training

GPTBot — OpenAI's training crawler. ClaudeBot — Anthropic's training crawler. Google-Extended — Google's training token (a preference token, not a crawler): blocking it opts out of Google AI training without touching normal Google Search. Applebot-Extended — Apple's training token, same idea for Apple Intelligence training. CCBot — Common Crawl, whose datasets feed many AI models. Meta-ExternalAgent — Meta's AI training crawler. Bytespider — ByteDance's crawler, widely blocked over training use.

Search and answer crawlers — block these and AI search may stop citing you

OAI-SearchBot — OpenAI's search index crawler, behind ChatGPT search results. PerplexityBot — Perplexity's search crawler. Claude-SearchBot — Anthropic's search crawler.

User-triggered fetchers — block these and assistants cannot open your pages on request

ChatGPT-User — fetches a page when a ChatGPT user asks about it. Claude-User — the same, for Claude.

Optional extras

One path to keep open for training (for example a press kit) Your sitemap address, if you have one

3. Your robots.txt

The shared token, honestly

The signal comes from an IETF Internet-Draft dated October 30, 2026, by Anthropic's Kevin Kelley. In your file it looks like this:

User-Agent: __usage_ai-training
Disallow: /

The timelines the companies have given: Anthropic has already begun implementing it. OpenAI says “the coming months.” Microsoft starts February 1, 2027. Google aims for 2027, and says it will read a token opt-out as a Google-Extended opt-out where Google-Extended is not named. Two cautions, stated plainly: the draft is an individual submission — not an IETF-endorsed standard — and it expires on May 3, 2027; and a User-Agent: * group does not count, which is why this builder always writes the token as its own group. We checked this claim too — see its Claim Check verdict.

What robots.txt cannot do

robots.txt is a request, not a lock. Honest crawlers honor it; dishonest scrapers ignore it, and nothing in the file stops them. It does not remove content that has already been collected into datasets or trained into models. And it is not access control — anything a crawler can read, a person can read. Treat it as stating your preference on the record, which is exactly what the companies above have agreed to respect.

How to use the file

Save it as robots.txt at the root of your domain, so it answers at https://yoursite.com/robots.txt. If you already have a robots.txt with rules for other bots or paths, merge — do not replace blindly: keep your existing groups and add the groups this builder made. Then visit your own /robots.txt address to confirm it is live, and re-check it after any site redesign.

For the record, bizn.org's own choice is the opposite of most of these settings: our robots.txt allows every crawler, because this site is built to be read by agents as well as people. Your site, your call — this builder exists so the call is yours to make, in one sitting, without learning the syntax.