XML Sitemaps in 2026: What the Spec Says, What Google Ignores, and What AI Crawlers Actually Do

An XML sitemap is a discovery file, not an indexing instruction. It tells search engines which URLs you consider worth crawling and, through lastmod, when each one last changed in a way that matters. Google states the boundary plainly: a sitemap "doesn't guarantee that all the items in your sitemap will be crawled and indexed" (Search Central, sitemap overview, verified 2026-08-23). Everything else people believe about sitemaps follows from misreading that one sentence.

This guide covers the protocol as written, the parts of it Google and Bing quietly ignore, every ceiling that binds at scale, the four submission methods that still work now that the ping endpoint is dead, and what the documentation actually says about AI crawlers. Where a claim is measured rather than cited, it says so and gives the date.

What an XML sitemap actually does

An XML sitemap does three things for a search engine: it lists the URLs you want crawled, it optionally reports through <lastmod> when each URL last changed significantly, and it provides a discovery path that does not depend on your internal linking. That is the whole job, and Google's sitemap documentation claims nothing beyond it.

Google describes submission itself as advisory. The build-and-submit documentation states that "submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site" (Search Central, build and submit a sitemap, page last updated 2026-07-08). No Google documentation claims a sitemap affects ranking. Discovery and ranking are separate systems, and a sitemap touches only the first.

This changes how you should debug a missing page. If a page is missing from the index, adding it to your sitemap is a weak intervention. The stronger questions are whether the page is reachable by internal links, whether it is canonical, whether it returns 200, and whether it is worth indexing at all. A sitemap surfaces candidates for crawling, and the rest of the decision happens elsewhere.

Do you need one at all?

Google publishes a size floor: you may not need a sitemap if your site is "about 500 pages or fewer", is well linked internally, and has no video, image, or news content that needs search visibility (Google Search Central, sitemap overview, page last updated 2025-12-10, verified 2026-08-23). Read that figure carefully. Google counts pages you think need to be in search results. That is not the same as URLs in your sitemap file, and not the same as every route your framework can generate, so a site with 500 indexable pages can legitimately sit under the floor while its sitemap holds far more entries.

Google names three situations where a sitemap does earn its place: a large site where new or updated pages are easy for crawlers to miss, a new site with few external links pointing at it, and a site with rich media or Google News content. A fourth case is operational rather than documented: a sitemap with honest lastmod values is a useful diagnostic surface during a technical SEO audit, because Search Console will report per-sitemap coverage and let you isolate a problem to one template.

Splitting sitemaps by content type is worth doing for that reason alone, even when you are nowhere near a size limit. One sitemap per template turns "some pages are not indexed" into "the product pages are not indexed", which is a question you can act on.

The spec has not changed since 2016. Google's behavior has.

The sitemaps.org protocol was last revised on 21 November 2016 and still defines three optional tags: <lastmod>, <changefreq>, and <priority> (sitemaps.org protocol 0.9, verified 2026-08-23). Google's documentation states, in its own words, "Google ignores <priority> and <changefreq> values" (build and submit a sitemap). Bing published the same position in July 2025: "Optional sitemap tags like changefreq and priority are ignored by Bing" (Bing Webmaster Blog, 2025-07-31).

Both facts are true at once, and the distinction matters in two concrete ways. First, these tags are not deprecated at the protocol level, so a sitemap containing them is still a valid sitemap, and any validator reporting them as errors is reporting its own opinion. Second, because they are valid but unused, removing them is a file-size optimization rather than a correctness fix. On a large sitemap that is a real saving. On a small one it changes nothing.

Google's stated reason for ignoring priority is worth quoting, because it explains why no amount of tuning will revive it. Gary Illyes wrote that "the priority element is a heavily subjective field and based on our internal studies, it generally doesn't accurately reflect the actual priority of a page" (Search Central Blog, 2023-06-26). The field was abandoned for being noise rather than removed for being harmful.

lastmod is the only optional tag that still earns its place

Google uses <lastmod> conditionally. Google Search Central states: "Google uses the <lastmod> value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate" (build and submit a sitemap, page last updated 2026-07-08, verified 2026-08-23). Verification runs against the page itself, so the value is a claim your site has to keep earning.

The significance test is documented rather than left to interpretation. Google states the value "should reflect the date and time of the last significant update to the page", and gives examples on both sides: a change to the main content, the structured data, or the links on the page is generally significant, while a change to the copyright date is not. A build pipeline that stamps every URL with the deploy timestamp fails this test on every page at once.

Illyes described the failure mode and its consequence in the same 2023 post: "if your page changed 7 years ago, but you're telling us in the lastmod element that it changed yesterday, eventually we're not going to believe you anymore." The penalty for dishonest lastmod is not a ranking action. It is that the field stops working for your whole site, including on the pages where it was accurate.

Partial adoption is explicitly permitted. Google sanctions leaving lastmod off URLs you are not confident about, which makes the correct policy straightforward: emit it where your CMS or build system knows the real content-change date, and omit it everywhere else.

Bing wants something stricter than Google does

This is the divergence almost nobody covers. Google's lastmod semantics are editorial: reflect significant change, omit when unsure. Bing's are operational: it asks for "standard ISO 8601 date formatting, including both the date and time", refreshed at least daily, and warns explicitly to "avoid setting lastmod to the time your sitemap was generated unless the content on that URL was actually updated" (Bing Webmaster Blog, 2025-07-31, authored by Bing principal product managers Fabrice Canel and Krishna Madhavan). Bing states it fetches a submitted sitemap immediately and revisits typically at least once per day.

A date-only lastmod satisfies Google and under-specifies for Bing. If Bing traffic matters to you, emit the full timestamp with an offset:

<url>
  <loc>https://example.com/guide</loc>
  <lastmod>2026-08-23T14:07:31+00:00</lastmod>
</url>

That value is valid W3C Datetime, valid for Google, and complete for Bing.

Every sitemap limit that binds, in one table

Every guide quotes 50,000 URLs. Very few quote the other nine ceilings, and several of those bite long before the URL count does.

Where the byte limit bites before the URL limit

The 50MB ceiling is measured uncompressed, and two configurations reach it well under 50,000 URLs.

Hreflang annotations expand quadratically. Google requires each <url> entry to list every alternate version including itself (localized versions, page last updated 2025-12-22). With n locales you write n child elements per entry across n entries, so annotation volume grows as n squared. A worked estimate, using my own assumptions rather than a Google figure: at 20 locales a single URL carries 20 <xhtml:link> elements, and at roughly 120 bytes per element the annotations alone cost about 2.4KB per entry. On that assumption you reach the 50MB ceiling near 20,000 entries, at about 40 percent of the permitted URL count. Measure your own output rather than trusting this arithmetic, because element length varies with URL length.

Image sitemaps carry up to 1,000 image entries per URL. A gallery template near that limit produces entries in the tens of kilobytes. The URL count stays trivial while the file size does not.

Gzip helps transmission but not compliance: the ceiling is defined on the uncompressed file. For reference, the sitemap on this site measured 6,015 bytes uncompressed on 2026-08-23 with this article included, which is roughly 0.01 percent of the 50MB ceiling. Its 52 URLs are about 0.1 percent of the 50,000 URL ceiling.

Sitemap index files, and the nesting rule that trips teams up

A sitemap index groups sitemap files, and it may not contain another sitemap index. Google's Search Console documentation names the resulting failure explicitly as the parse error "Incorrect sitemap index format: Nested sitemap indexes" (Search Console Help, Sitemaps report, verified 2026-08-23). Teams building very large sites reach for nesting as the obvious scaling move, and it fails silently until someone reads the report.

Three further rules govern index files (Search Central, manage sitemaps with index files, page last updated 2025-12-10):

• An index may contain up to 50,000 <loc> tags.
• Referenced sitemaps must be hosted on the same site as the index, unless cross-site submission is configured.
• Referenced sitemaps must sit in the same directory as the index or lower in the hierarchy. An index at /sitemap.xml can reference anything; an index at /blog/sitemap.xml cannot reference /shop/sitemap-products.xml.

That directory rule inherits from the protocol's path scoping: a sitemap at /catalog/sitemap.xml may only contain URLs under /catalog/. These are two different scoping rules and it is worth keeping them apart: the directory rule governs which sitemaps an index may reference, while path scoping governs which URLs a sitemap may list. Path scoping is lifted when the sitemap is declared in robots.txt or submitted through Search Console (sitemaps.org protocol), which is why a root-level declaration is the simplest posture for most sites.

Bing publishes the arithmetic these limits imply: 50,000 child sitemaps per index at 50,000 URLs each addresses 2.5 billion URLs from one index file, and multiple index files extend that to 2.5 trillion across a domain (Bing Webmaster Blog, 2025-07-31). Two cautions. The billion figure describes one index and the trillion figure describes a domain, and summaries frequently swap them. Bing's domain-level arithmetic also assumes no cap on index files, while Google limits Search Console to 500 index files per property, which puts the equivalent Google ceiling nearer 1.25 trillion.

How to submit a sitemap in 2026, now that ping is dead

The /ping?sitemap= endpoint is gone at both major engines. Google announced the deprecation on 26 June 2023, and the post now carries the banner "The sitemaps ping endpoint deprecation is complete" (Search Central Blog). Bing removed its equivalent thirteen months earlier, in May 2022 (Bing Webmaster Blog). Both cited spam: Google's stated reason was that "the vast majority of the submissions lead to spam".

Claude SEO measured both endpoints directly while this article was being written, requesting each with a populated parameter pointing at this site's live sitemap. On 23 August 2026, https://www.google.com/ping?sitemap=https://claude-seo.md/sitemap.xml returned HTTP 404 and the same request to https://www.bing.com/ping?sitemap= returned HTTP 410 Gone. If your CMS or deploy script still calls either URL, it is generating errors and accomplishing nothing. The code is harmless, and it is also pointless.

The four methods that do work

The Search Console API is the direct answer to "how do I automate submission now that ping is gone". The sitemaps resource exposes submit, delete, get, and list methods (Search Console API reference). Unlike ping it is authenticated, which is precisely why it survived the spam problem that killed the open endpoint.

For robots.txt, the directive rules are more permissive than most people assume (Search Central, robots.txt specifications, page last updated 2026-07-08). You can specify multiple sitemap fields with no stated limit. The value must be a fully qualified URL including protocol and host. The field name is case-insensitive while its value is case-sensitive. The field is not tied to any user agent, so position in the file is irrelevant. The sitemap does not have to be on the same host as the robots.txt file.

User-agent: *
Allow: /

Sitemap: https://claude-seo.md/sitemap.xml

The Sitemap directive is not part of the robots.txt standard

RFC 9309, the IETF Standards Track specification for the Robots Exclusion Protocol published in September 2022, does not define the Sitemap directive. Section 2.2.4, "Other Records", states only that crawlers "MAY interpret other records that are not part of the robots.txt protocol", and names Sitemaps as its example (RFC 9309). The Sitemap: line is a de facto convention that Google and Bing chose to honor, documented by sitemaps.org and by each engine, and sitting formally outside the standard. That distinction matters if you are reasoning about what a compliant crawler is obliged to do rather than what the major engines happen to do.

Sitemap extensions: what still works and what was removed

Image sitemaps are now a two-tag format. Only <image:image> and <image:loc> remain documented (Search Central, image sitemaps, page last updated 2025-12-10). Four tags were removed in Google's May 2022 sitemap extension cleanup: <image:caption>, <image:geo_location>, <image:title>, and <image:license>. The announcement stated the deprecated tags would "have no effect on indexing and search features after August 6, 2022" and required no action to remove (Search Central Blog, 2022-05-06). For licensing information Google directs publishers to IPTC metadata instead.

Video sitemaps kept more than people assume. The required set is <video:video> with <video:thumbnail_loc>, <video:title>, and <video:description> (maximum 2,048 characters), plus at least one of <video:content_loc> or <video:player_loc>, with content_loc recommended and required not to match the parent <loc> (Search Central, video sitemaps, page last updated 2026-05-20). The removed tags are <video:category>, <video:gallery_loc>, <video:price>, <video:tvshow>, and the autoplay and allow_embed attributes on player_loc. expiration_date, rating, view_count, and publication_date are all still supported, so check the current required-and-optional tables before removing anything.

News sitemaps follow a different limit entirely. A news sitemap may contain up to 1,000 <news:news> tags, not 50,000, and should include only articles created in the last two days (Search Central, news sitemaps, page last updated 2025-12-10). Google also asks publishers to update the existing file rather than create a new sitemap with each publication.

If you need more than one extension on the same URL, Google documents the pattern in a dedicated page covering namespace declaration on the urlset element, with the caveat that combining extensions "increases the file size of your sitemap significantly" (Search Central, combine sitemap extensions).

Do AI crawlers read your sitemap? An evidence audit

None of the three most-cited AI crawler operators documents using your XML sitemap. OpenAI, Anthropic, and Perplexity all publish crawler documentation, and as checked on 23 August 2026 none of them describes how URLs are discovered or mentions sitemaps at all. Sitemap advice has degraded fastest on this question, so what follows separates what is documented from what is assumed.

The primary crawler documentation for each operator:

• OpenAI operates GPTBot, OAI-SearchBot, and ChatGPT-User, and publishes their IP ranges. Its bot documentation contains no statement about how URLs are discovered and does not mention sitemaps (OpenAI bots documentation).
• Anthropic operates ClaudeBot, Claude-User, and Claude-SearchBot, and documents robots.txt and Crawl-delay support. It does not describe URL discovery and does not mention sitemaps (Anthropic crawler documentation).
• Perplexity documents PerplexityBot and Perplexity-User. Neither entry explains where the URLs come from, and the word sitemap does not appear on the page (Perplexity crawlers).

All three checked 2026-08-23. It is entirely plausible that these crawlers read sitemaps. It is simply not documented, and the honest position is to say so rather than to assert a mechanism nobody has published.

Google-Extended is not a crawler

Google states this precisely: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity" (Google crawler documentation, page last updated 2026-07-14). It is a robots.txt control token governing Gemini training and grounding, and Google adds that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Google-Extended cannot read your sitemap because Google-Extended does not fetch anything.

The documented path into Google's AI features

Google's guidance for generative AI features is that the entry requirement is ordinary Search eligibility: a page must be indexed and eligible to appear with a snippet. The same document states that "you don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities)" (Search Central, AI features guidance, page last updated 2026-07-10). That covers llms.txt directly, including the llms.txt file on this site, which exists as a convenience for crawlers that choose to read it rather than as a Google Search signal. Google Search Relations analyst Gary Illyes said at a Google Search Central Deep Dive event that Google does not support llms.txt and is not planning to, as reported by Search Engine Land on 2025-07-24. That is a reported remark rather than documentation, so treat the published guidance above as the stronger source. The wider question of what is documented about AI answer surfaces is covered in the guide to generative engine optimization.

Bing takes a more forward position, stating that sitemap freshness signals influence "how quickly updates are reflected in search results and AI generated answers" (Bing Webmaster Blog, 2025-07-31). That is a vendor statement from the engine that also owns IndexNow, published without methodology, and it should be attributed rather than repeated as established fact.

Where IndexNow fits, and where it does not

IndexNow does not accept sitemaps. Its documentation never mentions them: the protocol notifies participating engines about individual URLs that were added, updated, or deleted. The official FAQ states that IndexNow "is not designed for submitting every URL on your site at once" and recommends both: "For ongoing discovery and long-term indexing, use an XML sitemap" (IndexNow FAQ).

The division of labor the FAQ describes is the cleanest framing available. Sitemaps provide a complete inventory and cover pages that change infrequently or sit deep in the structure. IndexNow provides immediate notification of change. The FAQ even documents the backfill rule: use IndexNow for URLs that changed after setup, and sitemaps with accurate lastmod for content updated before it.

The participant list is Amazon, Bing, Naver, Seznam.cz, Yandex, and Yep (IndexNow FAQ, verified 2026-08-23). Google is not among them, and Google's documentation does not mention IndexNow. Google was reported in November 2021 as saying it would test and evaluate the protocol (Search Engine Journal), and has published nothing since. The defensible statement is that Google has neither adopted nor formally rejected it.

Crawl budget: what a sitemap actually changes

A sitemap does not increase your crawl budget. It changes how that budget is spent. Google lists keeping sitemaps up to date and including <lastmod> for updated content among its crawl budget management techniques, but claims no increase in capacity (Search Central, managing crawl budget, page last updated 2026-07-22).

The thresholds for who should care are higher than most people assume: large sites with more than one million unique pages changing roughly weekly, medium-or-larger sites with more than 10,000 unique pages changing daily, or sites with a large share of URLs reported as "Discovered - currently not indexed". Below that, crawl budget is usually the wrong thing to optimize.

Google also states that blocking URLs in robots.txt to free up budget does not work as intended: "Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit."

Reading the Search Console Sitemaps report

Google Search Console has no sitemap warnings tier. Its Sitemaps report has exactly three statuses: Success, Has errors, and Couldn't fetch (Search Console Help, verified 2026-08-23). There is no warnings tier to clear.

Worth noting for anyone searching for the old troubleshooting page: developers.google.com/search/docs/crawling-indexing/sitemaps/troubleshooting-sitemaps, the URL widely cited across the industry for sitemap error handling, returned HTTP 404 when I checked it on 2026-08-23. The Search Console Help page above is the current source.

Two behaviors in that documentation prevent unnecessary panic:

• One bad URL does not break the file. "Issues affecting individual URLs within a sitemap won't prevent Google from continuing to read the sitemap, as long as the sitemap can be fetched and read in general." URLs that parse cleanly are still queued.
• Fetch failures are retried. Google continues attempting a failed sitemap for a few days before giving up on that URL.

The documented causes of "Couldn't fetch" include some that have nothing to do with your XML: the sitemap being blocked by robots.txt, an unresolved manual action on the site, a 404 or server error, and low crawl demand for the sitemap. The manual action case is the one that wastes the most debugging time, because the file itself is perfect.

Auditing a real sitemap with one command

Every rule above is a mechanical check, which means a tool can run all of them against a live sitemap in one pass. Claude SEO is an MIT-licensed skill for Claude Code, currently at v2.2.4, published 2026-07-20 and verified against the repository on 2026-08-23. Its /seo sitemap command runs the validation pass in one step:

/seo sitemap https://example.com

Discovery runs first, reading the Sitemap: declarations in robots.txt and falling back to common paths when none resolves. Its operating rule is the part worth borrowing for any audit: a Sitemap: line in robots.txt is not proof that a sitemap works, so declared-but-broken sitemaps are preserved as findings instead of counted as successes.

It is one command inside the wider Claude SEO skill set, and the same check runs as one step of a full site audit. Findings come back ranked by severity rather than as an unordered list:

What it found on this site

Running the same checks against claude-seo.md on 2026-08-23, with this article already added, produced a mostly clean result and one genuine finding.

The file holds 52 URLs at 6,015 bytes, far inside both ceilings. Every URL is HTTPS and canonical, no deprecated tags are present, and the sitemap is declared in robots.txt. Then the Low-severity check fires: 44 of the 52 URLs carry the identical lastmod value of 2026-07-21, and the whole file contains only four distinct dates.

That is the exact pattern Google's guidance discounts. Those 44 pages did not all change significantly on the same day. The value reflects a site-wide edit, not per-page content change, so it fails the "consistently and verifiably accurate" test and provides no useful scheduling signal. The correct fix is per-page dates derived from real content changes, or omitting lastmod where the build cannot determine one. This site is publishing a sitemap that satisfies every hard rule and still under-uses the one optional tag that works. The entry for this article carries its own true publication date, which is the standard the other 44 need to meet.

What is proven, and what is folklore

Two things are widely repeated about sitemaps that no source supports: that their benefit has been measured, and that Google changed its sitemap guidance recently. Neither holds up.

There is no credible recent research quantifying sitemap benefit. While researching this article I looked for large-scale 2025 or 2026 studies with published methodology measuring the effect of sitemaps on indexation speed or coverage. I found none. Claims of the form "sitemaps get pages indexed X percent faster" circulate widely with no traceable upstream source. Treat any specific percentage you encounter as unsourced until someone shows you the methodology.

Google published nothing new about sitemaps in 2025 or 2026. Checked on 2026-08-23, the Search Central Blog carries no sitemap posts in either year, and the documentation changelog carries no sitemap entries for either year. The most recent substantive Google statement on lastmod remains the post from 26 June 2023. A 2026 sitemap policy change from Google is not something the public record supports.

That leads to a methodological point worth keeping. Several sitemap documentation pages carry recent "Last updated" stamps, including 2026-07-08 on the build-and-submit page. Those stamps are not backed by changelog entries, which suggests a documentation rebuild rather than a policy change. A "Last updated" date on a documentation page is not evidence that the guidance changed. Check the changelog before treating it as news.

What is solidly established is narrow and useful: the format and its limits, that priority and changefreq are ignored, that lastmod is used when it is trustworthy, that submission is a hint rather than a guarantee, and that the ping endpoint is gone. Build against those. The rest is either vendor positioning or repetition.

The short version

• List canonical, indexable, 200-status URLs only. Exclude noindexed, redirected, and non-canonical variants.
• Emit lastmod from real content-change dates, with time and offset. Omit it rather than guess.
• Drop priority and changefreq when file size matters, and otherwise ignore them.
• Declare the sitemap in robots.txt and submit it in Search Console. The two do different jobs.
• Split by template well before you approach any limit, because it makes coverage problems diagnosable.
• Automate submission through the Search Console API, not through a ping URL that returns 404.

The sitemap command, in short

git clone --depth 1 https://github.com/AgriciDaniel/claude-seo.git && bash claude-seo/install.sh

/seo sitemap https://example.com

/seo sitemap generate

PRIMARY SOURCES

Every engine-behavior claim above is cited to primary documentation. Each source was verified on the date shown.

1. Sitemaps XML format, protocol 0.9, sitemaps.org. Verified 2026-08-23.
2. What is a sitemap?, Google Search Central. Verified 2026-08-23.
3. Build and submit a sitemap, Google Search Central. Verified 2026-08-23.
4. Manage your sitemaps with sitemap index files, Google Search Central. Verified 2026-08-23.
5. Image sitemaps, Google Search Central. Verified 2026-08-23.
6. Video sitemaps, Google Search Central. Verified 2026-08-23.
7. Create a Google News sitemap, Google Search Central. Verified 2026-08-23.
8. How to combine sitemap extensions, Google Search Central. Verified 2026-08-23.
9. Sitemaps ping endpoint is going away, Google Search Central Blog. Verified 2026-08-23.
10. Spring cleaning: sitemap extensions, Google Search Central Blog. Verified 2026-08-23.
11. Sitemaps report, Google Search Console Help. Verified 2026-08-23.
12. robots.txt specifications, Google Search Central. Verified 2026-08-23.
13. Large site owner's guide to managing crawl budget, Google Search Central. Verified 2026-08-23.
14. Localized versions of your pages, Google Search Central. Verified 2026-08-23.
15. Optimizing for Google's generative AI features, Google Search Central. Verified 2026-08-23.
16. Google's common crawlers, Google. Verified 2026-08-23.
17. Search Console API: Sitemaps resource, Google. Verified 2026-08-23.
18. RFC 9309: Robots Exclusion Protocol, IETF. Verified 2026-08-23.
19. Keeping content discoverable with sitemaps in AI powered search, Bing Webmaster Blog. Verified 2026-08-23.
20. Spring cleaning: removed Bing anonymous sitemap submission, Bing Webmaster Blog. Verified 2026-08-23.
21. IndexNow FAQ, IndexNow. Verified 2026-08-23.
22. OpenAI bots, OpenAI. Verified 2026-08-23.
23. Does Anthropic crawl data from the web?, Anthropic. Verified 2026-08-23.
24. Perplexity crawlers, Perplexity. Verified 2026-08-23.
25. Google says normal SEO works for ranking in AI Overviews, and llms.txt will not be used (reported remark, not documentation), Search Engine Land. Verified 2026-08-23.
26. Google will be testing IndexNow (reported remark, not documentation), Search Engine Journal. Verified 2026-08-23.
27. Google Search Central Blog archive (checked for sitemap posts in 2025 and 2026), Google Search Central. Verified 2026-08-23.
28. Google Search documentation changelog (checked for sitemap entries in 2025 and 2026), Google Search Central. Verified 2026-08-23.

DANIEL AGRICI

Daniel Agrici builds Claude SEO, an MIT-licensed SEO toolkit for Claude Code, and maintains the sitemap validator described in this article. The sitemap findings here come from running those checks against live sites, including this one. Engine behavior is cited to primary documentation with the date it was verified. Claims about crawler internals that vendors do not publish are marked as undocumented rather than inferred.

Corrections and method. Corrections and questions are welcome through the GitHub repository issues tracker. This article was researched and drafted with AI assistance and is published under my name, which means the claims, the citations, and any error in them are mine. Every engine-behavior statement was checked against primary documentation on the verification date shown beside it, and first-hand measurements state the method so a reader can repeat them.

Report a correction // GitHub // YouTube // Community