WEBVTT

1
00:00:00.000 --> 00:00:03.860
Robots.txt is a public instruction file for compliant crawlers.

2
00:00:04.040 --> 00:00:11.280
It can reduce crawling, but it cannot protect private content, guarantee removal from search, or settle every question about AI use.

3
00:00:11.440 --> 00:00:13.080
Start with the outcome you want.

4
00:00:13.260 --> 00:00:16.040
Use robots.txt to control crawling.

5
00:00:16.220 --> 00:00:19.420
Use a crawlable noindex directive to control indexing.

6
00:00:19.760 --> 00:00:22.800
Use redirects or canonical tags for duplicate pages.

7
00:00:23.220 --> 00:00:26.420
Use authentication or network restrictions for anything private.

8
00:00:26.420 --> 00:00:29.800
Place the file at the lowercase root of each protocol and host.

9
00:00:30.040 --> 00:00:31.220
Keep the file simple.

10
00:00:31.460 --> 00:00:37.440
Use the supported fields: user agent, disallow, allow, and absolute sitemap URLs.

11
00:00:38.040 --> 00:00:40.580
Remember that the longest matching path wins.

12
00:00:41.060 --> 00:00:47.340
A specific crawler group does not inherit restrictions from the wildcard group, which is an easy way to create accidental access.

13
00:00:47.860 --> 00:00:50.520
Treat robots.txt as an HTTP endpoint too.

14
00:00:51.120 --> 00:00:53.460
A normal four-oh-four usually means no restrictions.

15
00:00:54.260 --> 00:00:56.360
Server failures can stop compliant crawling.

16
00:00:56.800 --> 00:01:03.200
Google also treats four-twenty-nine as a server error, so an unstable endpoint can cause more damage than a bad line of syntax.

17
00:01:03.740 --> 00:01:09.500
For AI systems, separate search crawlers, training crawlers, product controls, and user-triggered fetchers.

18
00:01:09.760 --> 00:01:12.320
There is no universal block-AI switch.

19
00:01:12.800 --> 00:01:19.780
Use each operator's current documentation, record the policy decision, and verify real requests in your logs.

20
00:01:19.780 --> 00:01:27.380
Before publishing a change, save the current response, test representative URLs for every affected crawler, and keep a rollback copy.

21
00:01:27.740 --> 00:01:32.320
The practical rule is simple: robots.txt expresses a crawl preference.

22
00:01:32.640 --> 00:01:35.900
It is not a lock, a removal tool, or a security boundary.

23
00:01:36.300 --> 00:01:39.200
Use the right control for the job, and test the public response.