How to find broken links from your sitemap.xml (and check every URL in it)

Your sitemap is supposed to be the list of pages you want search engines to index. Over time it drifts. Pages get deleted but stay in the sitemap, URLs get moved behind a redirect, someone adds noindex to a page that's still listed, and links inside your content start pointing at pages that no longer exist.

Google Search Console will tell you about some of this eventually, but it's slow and it only reports what Google happened to crawl. If you want to know today whether every URL in sitemap.xml actually works, you need to check them yourself.

This post covers three ways to do that, what each one is good for, and what the output looks like from a real run.

Option 1: curl in a loop (free, local)

For a small sitemap, a few lines of shell is enough:

curl -s https://www.example.com/sitemap.xml \ | grep -o '<loc>[^<]*' | sed 's/<loc>//' | head -50 \ | while read -r url; do curl -s -o /dev/null -L --max-time 15 \ -w "%{http_code} %{num_redirects} %{url_effective} $url\n" "$url" sleep 0.5 done 

Each line prints the final status code, how many redirects were followed, where the URL ended up, and the original URL. Pipe it into grep -v '^200 0' to see only the URLs that need attention.

This works, and for a one-off check it's honestly all you need. The catches:

• Sitemap indexes (a sitemap that lists other sitemaps) and .xml.gz sitemaps need extra handling. Many CMSs, including WordPress with Yoast or Rank Math, use an index by default.
• You only see the final hop. A 301 → 302 → 200 chain shows up as 200 2, without the URLs in between.
• It doesn't tell you about noindex or canonical tags, which is where sitemaps often contradict themselves.
• It only checks the pages in the sitemap, not the links inside those pages.
• Keep the sleep in. Firing hundreds of requests at once at a site, even your own, can trigger rate limiting or a firewall block.

Option 2: a desktop crawler or an open-source link checker

If you want more than status codes, there are good existing tools:

• Screaming Frog SEO Spider can crawl in list mode from a sitemap and reports redirects, canonicals and directives. The free version is limited to 500 URLs per crawl.
• lychee and linkchecker are open-source command-line link checkers. They're good for checking links inside pages or docs, and lychee fits well in CI.

These run on your own machine, so they're free (apart from Screaming Frog's paid licence above 500 URLs) and fast. The trade-off is that someone has to remember to run them, and scheduling them means keeping a machine or CI job around.

Option 3: a hosted sitemap link checker

If you want a spreadsheet of results, or a scheduled check you don't have to babysit, you can run it in the cloud. Full disclosure: I built a tool for this on Apify, the Sitemap Link Checker. It costs US$0.50 per 1,000 URLs checked, so compare that with Options 1 and 2 before you pick it. The rest of this walkthrough uses it because that's where the sample output comes from.

What it does differently from the script above:

• Reads sitemap indexes (including nested ones), .xml.gz files, plain-text sitemaps and the Sitemap: lines in robots.txt. If you only give it a homepage, it finds the sitemap itself.
• Records every redirect hop, not just the final one.
• Checks noindex (both the meta robots tag and the X-Robots-Tag header) and the canonical URL in the same request.
• Can optionally check the links found on each page, one level deep, and tells you which page each broken link was found on.

It's plain HTTP, with no browser. It respects robots.txt, including Crawl-delay, sends at most 2 requests per second per site by default, and backs off automatically when a site answers 429 or 503.

Step 1: Point it at a sitemap

Open the Actor, click Try for free, and switch to the JSON input. This is the exact input I used for the sample below:

{ "sitemapUrls": ["https://docs.apify.com/sitemap.xml"], "maxUrls": 30, "crawlLinks": true, "maxLinkUrls": 60, "sameDomainOnly": false } 

• sitemapUrls can be a normal sitemap, a sitemap index, a .gz file or a robots.txt URL. You can give several.
• maxUrls caps how many sitemap URLs are checked. The default is 50 so a first run is quick. Raise it once the output looks right.
• crawlLinks also checks the <a href> links found on those pages. This is how you find broken links inside your content rather than in the sitemap itself.
• maxLinkUrls caps those extra link checks, on top of maxUrls.
• sameDomainOnly: false includes outbound links to other sites. Leave it at true if you only care about your own pages.

That sitemap turned out to be an index pointing to 6 sitemaps. The run read the index and the first of those (727 URLs), checked 30 sitemap URLs plus 60 links found on those pages, and finished in about 46 seconds.

Step 2: Read the output

You get one row per URL. This is a real row from that run, a link found on one of the docs pages:

{ "url": "https://docs.apify.com/cli", "source": "page", "foundOn": "https://docs.apify.com/open-source", "statusCode": 301, "finalStatusCode": 200, "statusClass": "2xx", "isBroken": false, "finalUrl": "https://docs.apify.com/cli/docs", "redirectCount": 1, "redirectChain": [ { "url": "https://docs.apify.com/cli", "statusCode": 301, "location": "https://docs.apify.com/cli/docs" } ], "responseTimeMs": 1699, "contentType": "text/html; charset=utf-8", "noindex": false, "canonical": "https://docs.apify.com/cli/docs", "canonicalIsSelf": true, "error": null } 

The fields I'd look at first:

• isBroken: true when the final status is 4xx or 5xx, or the request failed outright (DNS, timeout, TLS error, redirect loop). Filter on this first.
• statusCode vs finalStatusCode: the first response and the one after all redirects. A sitemap URL with statusCode: 301 should usually be replaced with its finalUrl.
• redirectChain: every hop. Chains longer than one hop are worth flattening, especially after a site migration.
• foundOn: for links found inside pages, the page that contains the link, so you know where to fix it.
• noindex and canonicalIsSelf: a page that's in your sitemap but marked noindex, or whose canonical points somewhere else, is sending search engines mixed signals.

There's also a SUMMARY record in the run's key-value store. Here's the relevant part from the same run:

{ "urlsChecked": 90, "bySource": { "sitemap": 30, "page": 60 }, "statusCounts": { "2xx": 87, "3xx": 2, "4xx": 0, "5xx": 0, "error": 0 }, "finalStatusCounts": { "2xx": 89, "3xx": 0, "4xx": 0, "5xx": 0, "error": 0 }, "robotsBlocked": 1, "brokenCount": 0, "redirectedCount": 2, "redirectChainsOver1Hop": 0, "noindexCount": 0, "nonSelfCanonicalCount": 1, "avgResponseTimeMs": 680 } 

In this case there were no broken links at all, which is a good result for Apify's docs team. A clean report is a normal outcome, and that's the point of running it regularly. What it did find:

• 2 internal links that go through a redirect (/cli → /cli/docs, and an old SDK guide URL that 302s to the overview page). They work, but pointing the links at the final URL saves a hop.
• 1 outbound link, to x.com, that wasn't checked because x.com's robots.txt disallows it. It's marked robotsBlocked: true rather than being reported as broken.
• 1 outbound Discord invite link whose canonical points to a different URL, which is expected for a third-party page and nothing to fix.

Export to CSV or Excel and filter isBroken = true, or sort by redirectCount, to get a to-do list.

Step 3: Cost

That run checked 89 URLs that returned a response. The robots-blocked one isn't charged, and nor are requests that get no response at all, such as a DNS failure or timeout. At $0.0005 per URL plus a $0.00005 start fee, that comes to about $0.045. A 10,000-URL sitemap costs about $5. You can set Maximum cost per run in the run options, and the Actor stops cleanly when it reaches that limit.

Speed is set by the per-site rate limit, not by the tool. At the default 2 requests per second, 1,000 URLs on one site take a little over 8 minutes. If it's your own site, you can raise maxRequestsPerSecondPerHost.

Step 4: Automate it

The version that actually catches problems is a scheduled one:

1. Save your input as a task.
2. Add a schedule, for example weekly, or daily for a site that changes a lot.
3. Add a webhook or integration that sends you the SUMMARY record, and check brokenCount and redirectChainsOver1Hop.

You can also call it from CI after a deploy with Apify's API, using the normal asynchronous run endpoint for large sitemaps, then fail the build if brokenCount is above zero.

Caveats worth knowing (whichever option you pick)

• No JavaScript rendering. Plain HTTP checkers, including curl and this Actor, only see links that are in the HTML the server sends. Links added by JavaScript after the page loads need a browser-based crawler.
• Some servers lie to HEAD requests. That's why the Actor uses GET. If you write your own script with curl -I, expect a few false 405s or 404s.
• A 429 isn't a broken page. It means the site rate-limited you. Slow down and re-check those URLs rather than deleting them.
• robots.txt cuts both ways. If your own robots.txt blocks bots, a polite checker won't check those paths. Allow the checker's user agent for the paths you want checked.
• Only check sites you own or have permission to test, and keep the request rate gentle on sites that aren't yours.

Wrapping up

For a quick one-off check of a small sitemap, the curl loop above is enough. For a deeper crawl on your own machine, Screaming Frog (up to 500 URLs free) or lychee are good choices. When you want sitemap indexes, redirect chains, noindex and broken in-content links in one spreadsheet, on a schedule, a hosted checker saves you the plumbing.

If you try the Actor and something doesn't work, or there's a check you wish it did, open an issue on its Issues tab. I'd like to hear about it.