Methodology and crawl policy
What the Content Refresh Check can and cannot tell you, how it samples large sites, and how it decides what to check first.
What this tool cannot tell you
Without analytics or Search Console access, this tool cannot prove that a page has lost traffic, rankings, conversions, or business value. Every finding is a refresh opportunity — evidence that a page may be worth reviewing — never a claim about its performance.
Discovering and sampling pages
Sitemaps are discovered from robots.txt directives, then /sitemap.xml, then /sitemap_index.xml as a fallback — following sitemap indexes as needed. When a site has more eligible pages than the report's page limit, pages are sampled deterministically: a mix of the most recently modified pages, the oldest pages, and a stable, path-balanced sample of the rest, seeded from the sitemap URL itself. Re-running a scan on the same sitemap selects the same pages — sampling is disclosed on every report, never silent.
Freshness age bands
Age is a review signal, not proof of decay — it only contributes to prioritization when paired with a sufficiently confident date (structured datePublished/dateModified data first, then a <time> element, then sitemap lastmod as a lower-confidence fallback). Copyright/footer years are never treated as publication dates.
| Age since last reliable update | Signal |
|---|---|
| Under 12 months | No age-based opportunity |
| 12–24 months | Low-priority review |
| 24–36 months | Medium-priority review |
| Over 36 months | Higher-priority review |
How opportunities are ordered
Every page gets an internal ordering score built from weighted signals — used only to sort the list, never displayed as an objective quality percentage. Confidence modifies the contribution of each signal: confirmed evidence (a verified 4xx/5xx link) outranks uncertain heuristics (a possibly outdated year mentioned in a title).
| Signal | Maximum contribution |
|---|---|
| Confirmed broken outbound links | 30 |
| Reliable age and time-sensitive language | 25 |
| Missing or duplicated metadata | 20 |
| Possible topic overlap | 15 |
| Missing relevant author information | 5 |
| Thin extractable content | 5 |
Broken links, outdated references, and topic overlap
Outbound links are checked with a bounded HEAD request (GET fallback for servers that reject HEAD), with every redirect hop re-validated. A timeout or a 403/401 is never called a confirmed broken link — only a verified 4xx/5xx response is. Possibly outdated references (old-but-plausible years, version numbers, or time-sensitive language like "currently" or "latest" on an old page) are always labeled as candidates, never asserted facts — this tool maintains no database of current product versions. Topic overlap is detected with deterministic TF-IDF text similarity (no embeddings or LLM), requiring both title/heading overlap and body similarity above a conservative threshold, and is always described as possible overlap — similar pages can legitimately serve different audiences.
Crawl limits and etiquette
- Maximum 50 pages analyzed per report.
- Maximum 5 sitemap files fetched, including sitemap indexes.
- Maximum 5,000 sitemap entries parsed before sampling.
- Maximum 200 unique outbound links checked per report.
- Maximum 3 concurrent requests to the target site, 2 per external hostname for link checks.
Every initial target, sitemap URL, page URL, and redirect is validated against private, loopback, link-local, and cloud-metadata network ranges before being requested. The scanner identifies itself with a dedicated user agent, respects robots.txt, and caches results to avoid repeated traffic to the same site.
