Disclosure: we built and sell the Apify Actor this post mentions at the end, Sitemap Validator (publisher WrenTools). The post was written with AI assistance and reviewed by a person before publishing. Every output below was produced by one script, run against a small test site on our own machine whose every answer we know.
In short: the usual twenty-line sitemap checker (requests.get, ElementTree, loop over <loc>) reported 0 broken links on a test site that has 2. It found no URLs at all because of the XML namespace; once that was fixed it checked the index's child sitemaps instead of the pages; it could not parse the .xml.gz sitemap; requests.head called a working page broken and a broken one fine; and a rate limit turned 15 of 20 good pages into 429s. The fix: use the sitemaps.org namespace, recurse into sitemapindex files, check the gzip magic bytes, use GET and read r.history, and space your requests with a timeout.
The test site has a sitemap index pointing at two sitemaps:
/sitemap_index.xml
/sitemap-pages.xml / /pricing /old-pricing /features /team
/sitemap-posts.xml.gz /blog/a /blog/b /blog/c
Two of those eight pages are broken: /blog/c answers 404, and /team redirects (301) to /about-us, which answers 404. One more, /old-pricing, redirects to /pricing, which works but shouldn't be in a sitemap: a sitemap is meant to list final URLs.
Predict: a checker sends requests.head to each of the eight pages and counts every status of 400 or more as broken. Which pages does it report, and is it right? Decide before you look at the figure.
HEAD reports two pages, and gets one of them wrong: /pricing works, and /team, which is really broken, passes as a 301. Section 4 shows why. The traps in red come first, in the order a checker meets them.
1. Why does findall("url") return nothing?
Every sitemap declares the sitemaps.org namespace on its root, and ElementTree puts the namespace into every tag name:
root = ET.fromstring(requests.get(url, timeout=10).content)
print(root.tag)
print(len(root.findall("url/loc")))
print(len(root.findall("sm:url/sm:loc", {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"})))
{http://www.sitemaps.org/schemas/sitemap/0.9}urlset
0
5
Zero URLs means zero broken URLs, so this one fails quietly: the script prints a clean report. If a checker ever tells you a sitemap has no pages, suspect the namespace before you believe it.
2. A sitemap index is not a list of pages
Large sites split their sitemap and list the parts in a sitemapindex. Its entries also use <loc>, so a checker that collects every <loc> "works":
200 /sitemap-pages.xml
200 /sitemap-posts.xml.gz
urls checked: 2 | broken (>= 400): 0
It checked the two sitemap files, found them both fine, and never looked at a page. Check the root tag and recurse when it is sitemapindex. Keep a set of the sitemaps you've read, because an index that lists itself (or another index) would otherwise loop.
3. Why won't my .xml.gz sitemap parse?
When a server sends a gzipped sitemap as a file (Content-Type: application/x-gzip, no Content-Encoding header), requests hands you the compressed bytes as they are:
Content-Type: application/x-gzip | Content-Encoding: None
first bytes: 1f8b
ET.fromstring(r.content): ParseError: not well-formed (invalid token): line 1, column 0
after gzip.decompress: 3 urls
But a server can also send the same bytes with Content-Encoding: gzip, and then requests decodes them for you:
with Content-Encoding: gzip, first bytes: b'<?xml'
gzip.decompress(r.content): BadGzipFile: Not a gzipped file (b'<?')
So check the two magic bytes rather than the file name or the header; the bytes tell you which case you're in.
Here are fixes 1 to 3 together:
import gzip
import xml.etree.ElementTree as ET
import requests
NS = "http://www.sitemaps.org/schemas/sitemap/0.9"
def pages(sitemap_url, seen=None):
"""Every <loc> of a sitemap or a sitemap index, recursively, gzipped or not."""
seen = set() if seen is None else seen
if sitemap_url in seen:
return []
seen.add(sitemap_url)
body = requests.get(sitemap_url, timeout=10).content
if body[:2] == b"\x1f\x8b":
body = gzip.decompress(body)
root = ET.fromstring(body)
locs = [e.text.strip() for e in root.iter(f"{{{NS}}}loc")]
if root.tag == f"{{{NS}}}sitemapindex":
return [p for child in locs for p in pages(child, seen)]
return locs
On the test site it returns all eight pages.
4. HEAD or GET to check a link?
HEAD looks like the efficient choice: no body. Here are the first five pages both ways:
/ head 200 get 200
/pricing head 405 get 200
/old-pricing head 301 get 200 history 301 to /pricing
/features head 200 get 200
/team head 301 get 404 history 301 to /about-us
Two things go wrong with HEAD. Plenty of web apps only route GET, so a working page answers 405 Method Not Allowed and gets reported as broken. And requests.head does not follow redirects by default (requests.get does), so /team reads as a harmless 301 when the page it leads to is gone.
GET has the opposite problem: it follows the redirect and hands you the final 200, so /old-pricing looks fine. The answer is in r.history, the list of redirect responses on the way:
def check(url):
r = requests.get(url, timeout=10)
if r.history:
return f"redirects ({r.history[0].status_code}) to {r.url}, final {r.status_code}"
return str(r.status_code)
/ 200
/pricing 200
/old-pricing redirects (301) to /pricing, final 200
/features 200
/team redirects (301) to /about-us, final 404
/blog/a 200
/blog/b 200
/blog/c 404
That is the right answer: two broken pages, and one URL to replace with the one it redirects to.
5. Rate limits and timeouts
Our test site allows five requests a second, a small limit but a real kind. Twenty pages fired back to back:
20 back to back: 5 x 200, 15 x 429
spaced 250 ms, Retry-After honoured: 20 x 200, 0 x 429, 5.1 s
A checker that counts every 4xx as broken reports 15 of 20 good pages as broken. Space the requests, and when you do get a 429, wait for its Retry-After and try again. It's also the polite way to crawl your own site, which is often sharing a server with real visitors.
The last trap is the one that hangs a scheduled job. requests waits as long as the server does unless you pass a timeout:
no timeout: waited 5.0 s
timeout=2: ReadTimeout after 2.0 s
Our slow page answered after 5 seconds, and the script waited all 5. A real one that stalls can hold the job forever. Pass timeout= on every call (every snippet above does) and report a timeout as its own kind of failure rather than as a 404.
Questions people ask
How many URLs can one sitemap hold? The sitemaps.org protocol allows 50,000 URLs and 50 MB (52,428,800 bytes) uncompressed per file; past that, you split it and list the parts in an index, which is why trap 2 shows up on exactly the sites that matter.
Should I check the pages robots.txt blocks? If it's your own site, it's worth knowing that a blocked URL is in your sitemap at all: crawlers that obey robots.txt won't fetch it, so it shouldn't be listed.
Is a redirect in a sitemap an error? It isn't broken, but the sitemap should list the final URL, so treat it as something to fix. That's why the check above reports it separately.
Try it
If you'd rather not maintain this, our Sitemap Validator Actor does all of the above: it follows indexes, decompresses .gz sitemaps with a size guard, requests each page with GET and reports redirects with where they lead (redirectedTo), obeys robots.txt and waits 500 ms between requests by default. It also checks each entry against the sitemaps.org rules (<lastmod> format, <priority> range, duplicates, entries on another host). Status checking is off by default; with it on, each URL costs $0.0006, and each sitemap file $0.002. The example check a sitemap for broken links runs it from Python or Node, writes the problems to a CSV and exits with 1 when there are any, so it fits a scheduled job.
Which one has caught you: the namespace, the index, or the redirect to nowhere? Tell us in the comments.
Comments
Join the discussionNo comments yet. Be the first to comment.
