ProofRankBot

Identity

User agent: ProofRankBot/2.0 (+https://proofrank.net/bot). robots.txt token: ProofRankBot.

What it fetches

For each analysis: robots.txt, the sitemap, the homepage and up to 24 more pages from the sitemap and internal links, up to 25 addresses those pages link to or name as canonical or hreflang versions (a few kilobytes each, to check that they answer), up to 25 of the site's own images, scripts and stylesheets (the first 2 KB each, for their status and size), the favicon, the http:// address, one made-up address and the homepage again. One homepage request uses a desktop browser user agent to compare what visitors and crawlers receive, and one each uses the user agents of GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot, only when robots.txt allows that crawler, to check that the server lets AI crawlers in. It also reads /llms.txt and /.well-known/security.txt, checks up to 20 links to other sites (the first 2 KB each, after reading that site's robots.txt), and looks up the domain's public records: DNS (SPF, DMARC, CAA, DNSSEC, IPv6, nameservers), its registration expiry (RDAP) and its certificates in Certificate Transparency logs. When the browser is available, the page's text after scripts run is counted too. It does not submit forms or sign in.

Rules it follows

• robots.txt (RFC 9309): the ProofRankBot group, otherwise *. If robots.txt answers 401, 403, 429 or 5xx, or cannot be read, nothing else is fetched.
• noindex and nofollow in robots meta tags and X-Robots-Tag, and rel=nofollow links, for pages found through links.
• At most two requests at a time per site, a pause after each answer, Crawl-delay up to one second, and a slower pace after 429 or 503.

Blocking it

User-agent: ProofRankBot
Disallow: /

A site that blocks ProofRankBot is not scored.