# Check a sitemap for broken links in Python: five traps that hide them **Disclosure:** we built and sell the Apify Actor this post mentions at the end, [Sitemap Validator](https://apify.com/wrentools/sitemap-validator) (publisher WrenTools). The post was written with AI assistance and reviewed by a person before publishing. Every output below was produced by one script, run against a small test site on our own machine whose every answer we know. **In short:** the usual twenty-line sitemap checker (`requests.get`, `ElementTree`, loop over ``) reported **0 broken links** on a test site that has **2**. It found no URLs at all because of the XML namespace; once that was fixed it checked the index's child sitemaps instead of the pages; it could not parse the `.xml.gz` sitemap; `requests.head` called a working page broken and a broken one fine; and a rate limit turned 15 of 20 good pages into 429s. The fix: use the sitemaps.org namespace, recurse into `sitemapindex` files, check the gzip magic bytes, use GET and read `r.history`, and space your requests with a timeout. The test site has a sitemap index pointing at two sitemaps: ```text /sitemap_index.xml /sitemap-pages.xml / /pricing /old-pricing /features /team /sitemap-posts.xml.gz /blog/a /blog/b /blog/c ``` Two of those eight pages are broken: `/blog/c` answers 404, and `/team` redirects (301) to `/about-us`, which answers 404. One more, `/old-pricing`, redirects to `/pricing`, which works but shouldn't be in a sitemap: a sitemap is meant to list final URLs. **Predict:** a checker sends `requests.head` to each of the eight pages and counts every status of 400 or more as broken. Which pages does it report, and is it right? Decide before you look at the figure. ![A tree of the test site: sitemap_index.xml points at sitemap-pages.xml and sitemap-posts.xml.gz, which list the eight pages. Each page shows what requests.head says and what GET with r.history says. HEAD reports /pricing (405, a working page) and /blog/c (404), and misses /team, whose 301 leads to a 404. The index and both sitemaps are marked with traps 1 to 3.](https://cdn.hashnode.com/uploads/gql/6ad91735786cf66498f3c2f4/c2098702-0b99-4dc6-a032-d5b9e9da5e46.png) HEAD reports two pages, and gets one of them wrong: `/pricing` works, and `/team`, which is really broken, passes as a 301. Section 4 shows why. The traps in red come first, in the order a checker meets them. ## 1. Why does `findall("url")` return nothing? Every sitemap declares the sitemaps.org namespace on its root, and ElementTree puts the namespace into every tag name: ```python root = ET.fromstring(requests.get(url, timeout=10).content) print(root.tag) print(len(root.findall("url/loc"))) print(len(root.findall("sm:url/sm:loc", {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}))) ``` ```text {http://www.sitemaps.org/schemas/sitemap/0.9}urlset 0 5 ``` Zero URLs means zero broken URLs, so this one fails quietly: the script prints a clean report. If a checker ever tells you a sitemap has no pages, suspect the namespace before you believe it. ## 2. A sitemap index is not a list of pages Large sites split their sitemap and list the parts in a `sitemapindex`. Its entries also use ``, so a checker that collects every `` "works": ```text 200 /sitemap-pages.xml 200 /sitemap-posts.xml.gz urls checked: 2 | broken (>= 400): 0 ``` It checked the two sitemap files, found them both fine, and never looked at a page. Check the root tag and recurse when it is `sitemapindex`. Keep a set of the sitemaps you've read, because an index that lists itself (or another index) would otherwise loop. ## 3. Why won't my `.xml.gz` sitemap parse? When a server sends a gzipped sitemap as a file (`Content-Type: application/x-gzip`, no `Content-Encoding` header), requests hands you the compressed bytes as they are: ```text Content-Type: application/x-gzip | Content-Encoding: None first bytes: 1f8b ET.fromstring(r.content): ParseError: not well-formed (invalid token): line 1, column 0 after gzip.decompress: 3 urls ``` But a server can also send the same bytes with `Content-Encoding: gzip`, and then requests decodes them for you: ```text with Content-Encoding: gzip, first bytes: b' of a sitemap or a sitemap index, recursively, gzipped or not.""" seen = set() if seen is None else seen if sitemap_url in seen: return [] seen.add(sitemap_url) body = requests.get(sitemap_url, timeout=10).content if body[:2] == b"\x1f\x8b": body = gzip.decompress(body) root = ET.fromstring(body) locs = [e.text.strip() for e in root.iter(f"{{{NS}}}loc")] if root.tag == f"{{{NS}}}sitemapindex": return [p for child in locs for p in pages(child, seen)] return locs ``` On the test site it returns all eight pages. ## 4. HEAD or GET to check a link? HEAD looks like the efficient choice: no body. Here are the first five pages both ways: ```text / head 200 get 200 /pricing head 405 get 200 /old-pricing head 301 get 200 history 301 to /pricing /features head 200 get 200 /team head 301 get 404 history 301 to /about-us ``` Two things go wrong with HEAD. Plenty of web apps only route GET, so a working page answers `405 Method Not Allowed` and gets reported as broken. And `requests.head` does not follow redirects by default (`requests.get` does), so `/team` reads as a harmless 301 when the page it leads to is gone. GET has the opposite problem: it follows the redirect and hands you the final 200, so `/old-pricing` looks fine. The answer is in `r.history`, the list of redirect responses on the way: ```python def check(url): r = requests.get(url, timeout=10) if r.history: return f"redirects ({r.history[0].status_code}) to {r.url}, final {r.status_code}" return str(r.status_code) ``` ```text / 200 /pricing 200 /old-pricing redirects (301) to /pricing, final 200 /features 200 /team redirects (301) to /about-us, final 404 /blog/a 200 /blog/b 200 /blog/c 404 ``` That is the right answer: two broken pages, and one URL to replace with the one it redirects to. ## 5. Rate limits and timeouts Our test site allows five requests a second, a small limit but a real kind. Twenty pages fired back to back: ```text 20 back to back: 5 x 200, 15 x 429 spaced 250 ms, Retry-After honoured: 20 x 200, 0 x 429, 5.1 s ``` A checker that counts every 4xx as broken reports 15 of 20 good pages as broken. Space the requests, and when you do get a 429, wait for its `Retry-After` and try again. It's also the polite way to crawl your own site, which is often sharing a server with real visitors. The last trap is the one that hangs a scheduled job. requests waits as long as the server does unless you pass a timeout: ```text no timeout: waited 5.0 s timeout=2: ReadTimeout after 2.0 s ``` Our slow page answered after 5 seconds, and the script waited all 5. A real one that stalls can hold the job forever. Pass `timeout=` on every call (every snippet above does) and report a timeout as its own kind of failure rather than as a 404. ## Questions people ask **How many URLs can one sitemap hold?** The sitemaps.org protocol allows 50,000 URLs and 50 MB (52,428,800 bytes) uncompressed per file; past that, you split it and list the parts in an index, which is why trap 2 shows up on exactly the sites that matter. **Should I check the pages robots.txt blocks?** If it's your own site, it's worth knowing that a blocked URL is in your sitemap at all: crawlers that obey robots.txt won't fetch it, so it shouldn't be listed. **Is a redirect in a sitemap an error?** It isn't broken, but the sitemap should list the final URL, so treat it as something to fix. That's why the check above reports it separately. ## Try it If you'd rather not maintain this, our [Sitemap Validator](https://apify.com/wrentools/sitemap-validator) Actor does all of the above: it follows indexes, decompresses `.gz` sitemaps with a size guard, requests each page with GET and reports redirects with where they lead (`redirectedTo`), obeys robots.txt and waits 500 ms between requests by default. It also checks each entry against the sitemaps.org rules (`` format, `` range, duplicates, entries on another host). Status checking is off by default; with it on, each URL costs $0.0006, and each sitemap file $0.002. The example [check a sitemap for broken links](https://github.com/WrenTools/apify-actor-examples/tree/main/check-a-sitemap-for-broken-links) runs it from Python or Node, writes the problems to a CSV and exits with 1 when there are any, so it fits a scheduled job. Which one has caught you: the namespace, the index, or the redirect to nowhere? Tell us in the comments.