Robots.txt in 2026: The Complete Guide to Search and AI Crawlers

Robots.txt is a public, origin-specific text file that asks compliant crawlers which URL paths they may request. It can reduce unwanted crawling. It cannot protect private content, guarantee deindexing, consolidate duplicate URLs, or enforce a universal rule about AI use.

That distinction is the heart of robots.txt in 2026. Search crawlers, training crawlers, product control tokens, and user-triggered fetchers can all have different names and documented behavior. A good policy starts with the outcome you want, identifies the requesting system and purpose, then uses the least ambiguous control that system documents.

Critical warning: If content must stay private, require authentication or a network restriction. Do not advertise the secret path in a public robots.txt file.

The 60-second answer

• Put the file at the lowercase root path of each origin: https://example.com/robots.txt.
• Serve UTF-8 plain text, ideally with a successful 200 response and Content-Type: text/plain.
• Use User-agent, Disallow, and Allow to express crawl preferences. Add absolute Sitemap URLs when useful.
• Match specific crawlers carefully. A specific group does not inherit the wildcard group's restrictions.
• Use noindex for index control, redirects or canonical tags for duplicates, and authentication for privacy.
• Separate AI search, training, product-use, and user-triggered access decisions. There is no reliable universal “block AI” switch.
• Test representative URLs, inspect the public response, and keep a rollback copy.

// LISTEN

A faster summary in Daniel's voice

Press play to hear the main decisions. The transcript follows the audio and can also be used to seek.

0:00Robots.txt is a public instruction file for compliant crawlers.0:04It can reduce crawling, but it cannot protect private content, guarantee removal from search, or settle every question about AI use.0:11Start with the outcome you want.0:13Use robots.txt to control crawling.0:16Use a crawlable noindex directive to control indexing.0:20Use redirects or canonical tags for duplicate pages.0:23Use authentication or network restrictions for anything private.0:26Place the file at the lowercase root of each protocol and host.0:30Keep the file simple.0:31Use the supported fields: user agent, disallow, allow, and absolute sitemap URLs.0:38Remember that the longest matching path wins.0:41A specific crawler group does not inherit restrictions from the wildcard group, which is an easy way to create accidental access.0:48Treat robots.txt as an HTTP endpoint too.0:51A normal four-oh-four usually means no restrictions.0:54Server failures can stop compliant crawling.0:57Google also treats four-twenty-nine as a server error, so an unstable endpoint can cause more damage than a bad line of syntax.1:04For AI systems, separate search crawlers, training crawlers, product controls, and user-triggered fetchers.1:10There is no universal block-AI switch.1:13Use each operator's current documentation, record the policy decision, and verify real requests in your logs.1:20Before publishing a change, save the current response, test representative URLs for every affected crawler, and keep a rollback copy.1:28The practical rule is simple: robots.txt expresses a crawl preference.1:33It is not a lock, a removal tool, or a security boundary.1:36Use the right control for the job, and test the public response.

AI-generated audio using Daniel's authorized cloned voice at 1.1x speech speed. Soft background music is looped at 5% linear gain with fade in and fade out. Duration: 1:40.

This guide uses RFC 9309, current crawler-operator documentation, peer-reviewed research, and a reproducible audit of claude-seo.md. Volatile bot details were last verified on August 24, 2026.

Choose the outcome before the directive

Robots.txt becomes dangerous when it is used as a substitute for a different control. Choose the desired outcome first.

Desired outcome	Primary control	Why
Reduce compliant crawler requests	robots.txt	This is the protocol's direct purpose.
Keep a page out of search results	Crawlable noindex meta tag or X-Robots-Tag	The crawler must fetch the resource to read the directive.
Remove a deleted URL	404 or 410, with internal links and sitemap entries removed	A robots block can stop a crawler from seeing the removal status.
Consolidate duplicates	Redirect or rel="canonical", plus consistent links and sitemap URLs	Robots.txt is not a canonical signal.
Protect private or staging content	Authentication, IP restriction, or another access control	Robots.txt is public and voluntary.
Reduce abusive bot traffic	WAF rules, rate limits, verified identity, and logs	A hostile client can ignore the file or spoof a user-agent string.
Stay eligible for an AI search surface	Allow the operator's documented search crawler	Eligibility does not guarantee citation, ranking, or referral traffic.
Express a model-training preference	Use the operator's documented training crawler or product control	The control is operator-specific and does not govern every existing copy.

The first four distinctions align with Google's current documentation for robots crawl control, noindex, and canonicalization. They are separate systems, even when they affect the same URL.

Where robots.txt must live

The file applies to one protocol, host, and port. Rules do not automatically transfer between http and https, a root domain and a subdomain, or different ports.

Resource	Governing file
https://example.com/page	https://example.com/robots.txt
http://example.com/page	http://example.com/robots.txt
https://shop.example.com/page	https://shop.example.com/robots.txt
https://example.com:8443/page	https://example.com:8443/robots.txt

The path must be lowercase and at the root. /Robots.txt, /seo/robots.txt, and a file on a different host do not govern the target origin. RFC 9309 specifies UTF-8 content and a text/plain media type. Google's creation guide also documents this origin scope and says search crawlers discover the file automatically.

Robots.txt is public by design. Anyone can open it. Listing /client-contracts/, /backups/, or /staging-admin/ can disclose paths without preventing a noncompliant visitor from requesting them.

Minimal syntax that survives maintenance

A fully open site can use a small file with a sitemap reference:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

An absent or valid empty file also means there are no crawl restrictions for Google. The explicit version is useful when you want a visible policy and sitemap location. Allow: / is usually redundant when no Disallow rule competes with it.

To block a directory while allowing a deeper public section:

User-agent: *
Disallow: /reports/
Allow: /reports/public/

To match PDF URLs that end in .pdf:

User-agent: *
Disallow: /*.pdf$

The important fields are:

• User-agent: starts a group and identifies the crawler token.
• Disallow: asks the matching crawler not to request paths matching the value.
• Allow: permits a more specific path inside a broader blocked area.
• Sitemap: supplies an absolute sitemap URL. In Google's implementation, it is not scoped to the preceding user-agent group.
• #: begins a comment.

Google documents support for User-agent, Allow, Disallow, and Sitemap. It ignores unsupported fields such as crawl-delay. Other operators can support extensions, so never assume an extension is portable.

Paths are case-sensitive. /Shop/ and /shop/ are different. Matching includes the URL path and can include the query string. A percent-encoded character can also change the sequence being matched. The * wildcard matches any sequence of characters, and $ anchors the match to the end.

How rule matching really works

Consider this group:

User-agent: *
Disallow: /shop/
Allow: /shop/guides/
Disallow: /shop/guides/private.pdf

Requested path	Result	Winning rule
/shop/item	Blocked	/shop/
/shop/guides/start	Allowed	/shop/guides/ is the longer match
/shop/guides/private.pdf	Blocked	The exact, longer Disallow wins
/Shop/item	Allowed	Paths are case-sensitive

The parser first selects the applicable user-agent group, then finds the most specific rule for the requested URL. User-agent token matching is case-insensitive. URL path matching is case-sensitive. The longest matching path wins. If equivalent Allow and Disallow patterns conflict, RFC 9309 says Allow should win.

Rule order is not a general “first match wins” or “last match wins” system. Length decides specificity. Order can still affect human comprehension, so keep related rules together and put broad intent before narrow exceptions.

Separate groups with the same user-agent are combined under RFC 9309 and Google's parser. This means splitting one crawler across distant parts of a long file can produce a valid but hard-to-audit combined policy.

The specific-group inheritance trap

This is one of the most consequential maintenance errors in modern robots files:

User-agent: *
Disallow: /private/

User-agent: Bingbot
Crawl-delay: 5

Once Bingbot matches its specific group, the wildcard group's restriction is not a fallback. If Bingbot should also avoid /private/, repeat the shared rule:

User-agent: Bingbot
Disallow: /private/
Crawl-delay: 5

User-agent: *
Disallow: /private/

Bing documents both specific group behavior and its own support for crawl-delay in robots.txt guidance. Google does not support crawl-delay. Anthropic currently documents a nonstandard crawl-delay implementation for its crawlers. The directive therefore has to be evaluated per operator.

The risk grows when a file contains ten or twenty named AI groups. A future editor can add a wildcard restriction that looks global while every matching named group continues to use its own rules. The safest pattern is to create specific groups only when their policy genuinely differs. If they need a shared restriction, repeat it and test it.

HTTP status, redirects, caching, and size

A robots policy is also an HTTP endpoint. Its failure mode can matter more than its contents.

Situation	RFC 9309 baseline	Google's documented behavior
Successful 2xx response	Parse usable rules	Parses valid rules and ignores invalid lines
Redirect	Follow at least five consecutive redirects	Follows at least five; too many are treated like an unavailable file
Most 4xx responses	Treat the file as unavailable; crawler may access resources	Most mean no crawl restrictions
429 Too Many Requests	It is in the RFC's 4xx unavailable class; crawler may access resources	Google separately treats 429 as a server error
5xx or network failure	Treat the file as unreachable and assume complete disallow	Google initially stops crawling
Cache	Normally no more than 24 hours unless unreachable	Generally cached for up to 24 hours, adjusted by availability and cache headers
File size	Parser must support at least 500 KiB	Google ignores content after 500 KiB

RFC 9309 separates unavailable from unreachable. All 400 to 499 responses, including 429, are unavailable under the standard, so a crawler may access resources. 500 to 599 responses and network failures are unreachable, so the RFC baseline is complete disallow.

Google's robots.txt specification adds a product-specific exception: it treats 429 Too Many Requests as a server error. For a server error, Google stops crawling for the first 12 hours. It may then use the last good cached file for up to 30 days, followed by behavior that depends on site availability.

That creates a counterintuitive lesson: a valid 404 Not Found usually tells Google there are no robots restrictions, while a persistent 500 or 429 can slow or stop crawling. Do not put the endpoint behind an unstable application route if a simpler static response is available.

Robots.txt does not control indexing

Four myths cause most robots incidents.

Myth 1: Disallow removes a URL from Google

It does not guarantee removal. If Google discovers a blocked URL through links or other signals, the URL can still appear without a content snippet. Google cannot crawl the blocked page to understand its content.

Myth 2: Noindex works inside robots.txt

Google does not support noindex as a robots.txt field. Put a meta robots tag in crawlable HTML or send an X-Robots-Tag HTTP header for HTML or non-HTML resources.

Myth 3: Blocking a page and adding noindex is safer

The combination can prevent Googlebot from seeing noindex. If the goal is deindexing, let the crawler fetch the resource until the directive is processed.

Myth 4: Blocking duplicate pages consolidates their signals

Robots.txt is not a canonical signal. A block can hide a page-level canonical tag. Use a redirect when the duplicate should disappear, or use rel="canonical", consistent internal linking, and a clean XML sitemap when both URLs must exist.

Use this diagnostic shortcut:

Need no crawling?                 robots.txt
Need no indexing?                crawlable noindex
Need removal because it is gone? 404 or 410
Need duplicate consolidation?    redirect or canonical
Need privacy?                    authentication

Crawl budget without the folklore

Most sites do not need a crawl-budget strategy. Google's large-site crawl budget guide is aimed mainly at sites near one million moderately changing pages, sites with roughly 10,000 rapidly changing pages, or sites with many URLs stuck in “Discovered, currently not indexed.”

Google separates crawl capacity from crawl demand. Server health, duplicate URL spaces, faceted navigation, parameter traps, low-value URLs, and changing content can influence what gets requested. Robots.txt can stop compliant Googlebot requests to URL spaces that should never be crawled, but it should not disguise broken architecture.

Blocking URLs also does not promise that capacity will move to preferred pages. Google says reallocation is relevant only when the site was already constrained by crawl capacity. A temporary robots block is therefore a poor routine “priority boost.” Fix discovery, internal links, duplicate generation, server performance, and sitemap quality first. Use the technical SEO workflow to evaluate the whole crawl system, not one file in isolation.

AI crawlers in 2026: classify before choosing

“AI bot” is not one purpose. Current operator documentation distinguishes automatic search crawlers, training crawlers, product-use controls, and user-triggered page requests.

The following matrix was last verified on August 24, 2026. Operator names and policies can change, so recheck each linked primary source before production edits.

Purpose	Operator token	Documented robots behavior	Publisher tradeoff
ChatGPT search	OAI-SearchBot	OpenAI documents an independent robots choice	Search-answer eligibility versus opt-out
OpenAI training	GPTBot	OpenAI documents an independent robots choice	Training preference, separate from search
ChatGPT user action	ChatGPT-User	OpenAI says robots rules may not apply	Use access controls if retrieval must be prevented
Claude training	ClaudeBot	Anthropic documents robots support	Training preference
Claude search	Claude-SearchBot	Anthropic documents robots support	Search visibility versus restriction
Claude user action	Claude-User	Anthropic documents robots control	User-directed visibility versus restriction
Perplexity search	PerplexityBot	Perplexity documents robots support	Search-result eligibility; current docs say this is not a training crawler
Perplexity user action	Perplexity-User	Perplexity says it generally ignores robots.txt	Access control is needed for enforcement
Google AI use control	Google-Extended	Product control applied to data crawled by existing Google user agents	Training and some grounding choice, with no Google Search ranking effect
Apple search and discovery	Applebot	Apple documents robots support	Search visibility and documented downstream AI uses
Apple training control	Applebot-Extended	Product control, not a separate crawler	Training opt-out without blocking Apple search
Amazon general product use	Amazonbot	Amazon documents robots support but not crawl-delay support	Product improvement and possible training use versus restriction
Amazon search	Amzn-SearchBot	Amazon documents robots support	Search discovery versus restriction
Amazon user action	Amzn-User	Amazon says it may not follow all robots directives because requests can be user initiated	Use access control when retrieval must be prevented
Common Crawl dataset	CCBot	Common Crawl documents robots support	Future collection versus open-dataset distribution

OpenAI also notes that when search and training are both allowed, it may reuse one crawl for both purposes. Its documentation says changes may take about 24 hours to be reflected. ChatGPT-User, however, represents a user's request, not automatic search crawling.

Anthropic's current page documents three distinct tokens and publishes crawler IP information. Perplexity explicitly distinguishes PerplexityBot from its user-triggered fetcher. Google says Google-Extended has no separate HTTP user-agent string, so you should not expect that literal token to appear in server logs as an independent crawler.

These are three defensible policy profiles:

1. Discovery-first: allow documented search and training crawlers, then monitor logs and outcomes.
2. Search-visible, training-restricted: allow documented search crawlers while disallowing documented training crawlers or product controls.
3. Access-restricted: express preferences in robots.txt, then enforce sensitive access through authentication, verified WAF rules, rate limits, or network controls.

There is no universally correct profile. The decision belongs to the publisher and should distinguish discovery, model development, user-directed retrieval, and private access.

Safe AI policy examples

An open discovery policy can stay minimal:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

A publisher that currently wants OpenAI and Anthropic search discovery but not their documented training crawlers might use:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

That example does not block OAI-SearchBot or Claude-SearchBot, because they fall through to the open wildcard group. It also does not guarantee that user-triggered fetchers will be blocked. If the site later adds a wildcard restriction, every specific group must be retested for the inheritance trap.

Avoid copying enormous bot lists from an undated blog post. Each extra group creates maintenance cost. Keep a dated policy register with the token, purpose, operator source, desired rule, and last verification date.

What independent evidence says about compliance

RFC 9309 describes rules that crawlers are requested to honor. It explicitly says robots.txt is not a form of access authorization. Operator documentation tells you what a service says it does. Independent studies test behavior under bounded conditions. Those are different evidence types.

A peer-reviewed study by Kim and colleagues, published at IMC 2025, observed 130 self-declared bots over 40 days under controlled robots conditions. The open manuscript reports uneven compliance, with stricter directives followed less consistently. The study's institutional setting, time window, and reliance on self-declared identities limit universal attribution.

A July 2026 preprint tested ten AI assistants across 200 controlled trials and four allow or disallow conditions. It found substantial variation between fetching robots.txt, presenting an identifiable user agent, retrieving controlled content, and exposing that content in a visible answer. This is a preprint, not a peer-reviewed consensus, and it is a snapshot of tested systems at that time.

Neither study supports a timeless claim that a named operator always obeys or always ignores robots.txt. Together, they support a practical control stack:

Preference:   robots.txt and operator-specific controls
Verification: Search Console, server logs, current IP sources, DNS checks
Enforcement:  authentication, WAF rules, rate limits, network controls

User-agent strings can be spoofed. For security or billing decisions, verify identity using operator-published IP sources or documented reverse and forward DNS checks. Google's crawler verification guide explains the DNS process and publishes address ranges. Treat any copied IP list as volatile data, not permanent article content.

Emerging AI-use signals are not universal controls

Several 2026 initiatives try to express preferences that the basic robots protocol was not designed to carry.

• RFC 9969 records the 2026 IAB Workshop on AI-CONTROL. It is informational, not an Internet Standards Track specification.
• RSL 1.0 is an industry licensing specification with a License: discovery field and separate licensing data. It is not part of RFC 9309.
• Cloudflare Content Signals is a vendor-led vocabulary for search, AI input, and AI training preferences. Cloudflare describes preferences, not a universal technical barrier.
• Web Bot Auth is experimental work for cryptographic bot identity. It is not yet universal, and not every request is signed.

These ideas may become useful layers. They should not be sold as universally understood robots directives today. Google's robots parser documentation continues to list User-agent, Allow, Disallow, and Sitemap as supported fields.

CMS and framework implementation notes

The editing interface is less important than identifying the source of truth.

• Shopify: the platform says its defaults are appropriate for most stores. robots.txt.liquid customization is supported, but Shopify warns that errors can cause traffic loss. Edit the template, not a generated response, and preserve platform variables.
• WordPress: a site can expose a virtual default file or a physical root file. Plugins, hosting layers, and server files can compete. Check the public response and identify which layer generated it before following version-specific UI instructions.
• Next.js and similar frameworks: generate one root response, confirm production headers, and test the exact deployed host. Keep staging rules environment-specific so they cannot leak into production.
• CDN or edge platforms: check whether the route is cached, redirected, challenged, or rate-limited. A correct repository file does not prove the public endpoint is correct.

For generated sites, change the generator or template. Hand-editing generated output creates a fix that disappears on the next build.

Safe rollout and recovery checklist

Before the change

• Save the current public body, response headers, retrieval timestamp, byte size, and checksum.
• Write down the exact crawler tokens and URL patterns affected.
• Identify the source file, generator, framework route, CDN rule, and deployment path.
• Test critical page templates and assets, including CSS, JavaScript, images, APIs, and sitemaps.
• Check whether any named group will stop inheriting wildcard intent.
• Prepare the previous file for immediate rollback.

After the change

• Confirm the public endpoint returns the intended status, content type, encoding, and body.
• Test representative allowed and blocked URLs for every affected crawler group.
• Check Google's robots.txt report in Search Console and use URL Inspection for representative pages.
• Review server logs after crawler caches have had time to refresh.
• Verify crawler identity before turning a log entry into a security claim.
• Recheck after migrations, CDN changes, framework upgrades, or operator-policy updates.

Google generally caches robots.txt for up to 24 hours, but availability and cache headers can change that timing. A successful public deployment does not mean every crawler has refreshed immediately.

A first-hand audit of claude-seo.md

On August 24, 2026, I fetched https://claude-seo.md/robots.txt and compared it byte for byte with the local website source. This was a read-only audit. I did not inspect Search Console, CDN request logs, or verified crawler logs, and I did not change the live file.

Check	Observed result
Final HTTP status	200
Content type	text/plain; charset=utf-8
Cache control	public, max-age=0, must-revalidate
Body size	638 bytes
SHA-256	7e371089d245767e3ebbdf2b8ca3c98d38fae18debbf84252d27811a0ce0ba58
Live versus local source	Byte-for-byte match

The endpoint is healthy at the transport layer. Every listed group currently has Allow: /, so no listed compliant crawler is restricted. That looks consistent with an open documentation and open-source site.

The longer file still creates maintenance risk. It includes specific allow-only groups for Googlebot, Bingbot, GPTBot, ClaudeBot, Claude-SearchBot, Claude-User, OAI-SearchBot, ChatGPT-User, Google-Extended, PerplexityBot, and an anthropic-ai token. If a future editor adds Disallow: /private/ only to User-agent: *, these named groups will not inherit that restriction.

The list also combines different concepts. Google-Extended is a product control token, not a separate requesting user agent. ChatGPT-User is user-triggered, and OpenAI says robots rules may not apply. Anthropic's current documentation lists ClaudeBot, Claude-SearchBot, and Claude-User; it does not list anthropic-ai. I found no official deprecation statement, so the accurate label is “not verified in current documentation,” not “deprecated.”

For the site's current all-open behavior, this shorter file is easier to maintain:

# claude-seo.md robots.txt
# Open crawling policy. Last reviewed 2026-08-24.

User-agent: *
Disallow:

Sitemap: https://claude-seo.md/sitemap.xml

That recommendation is an article finding, not a production change. Before editing the live policy, the publisher should separately decide its search-discovery and model-training preferences, inspect actual logs, test the proposed file, and prepare rollback.

DANIEL AGRICI

This guide prioritizes standards bodies and crawler operators, then uses peer-reviewed research and clearly labeled preprints for independent evidence. Competitor articles informed topic-gap analysis but were not used as authority for technical claims. The live claude-seo.md case study records its date, limitations, response metadata, body size, and checksum so another reviewer can reproduce it.

Written and researched by Daniel Agrici, founder of Claude SEO. Technical review focuses on source fidelity, parser behavior, search-control boundaries, and claims that can change as operators update their systems. If you spot an outdated bot name or a source that has changed, open a correction issue with the URL, retrieval date, and correction requested.

For the broader implementation context, continue with the complete XML sitemap guide, technical SEO with Claude Code, or the generative engine optimization guide. You can also inspect Claude SEO's machine-readable publishing summary in llms.txt.

Report a correction // GitHub // YouTube // Community