- Home
- Methodology
How we measure
Every number on this site comes from our own crawler. Here is what it does and where its limits are.
Samples
| Market | Rule | Sites | Crawled | Homepage parsed |
|---|---|---|---|---|
| Global | Top 25,000 of the Tranco list | 100,000 | 100,000 | 68,167 |
| Netherlands | Top .nl domains | 17,173 | 17,173 | 8,376 |
| Japan | Top .jp domains | 16,643 | 16,643 | 12,356 |
| Italy | Top .it domains | 15,094 | 15,094 | 7,300 |
| United Kingdom | Top .uk domains | 30,000 | 30,000 | 11,322 |
| Spain | .es domains | 5,416 | 5,416 | 4,244 |
| Latin America | .mx .ar .cl .pe .uy .com.co … and top .br | 26,629 | 26,629 | 19,369 |
What we check on each site
- DNS: A/AAAA, NS, MX, TXT (SPF and SaaS verification), DMARC, CAA, DNSSEC, BIMI, MTA-STS.
- TLS: certificate authority, validity, protocol version — with a direct handshake.
- Homepage: technologies (500 signatures), SEO tags, Schema.org, page weight, third parties.
- Response headers: security headers, compression, HTTP version, cookies.
- robots.txt (incl. AI crawlers), sitemap.xml, llms.txt, security.txt, ads.txt.
- One visit per site and file; robots.txt is respected; bot challenges are recorded, not bypassed.
Sources
Tranco947W2
Date12 Nov 2026
IANA1,436 zones
Lists: Tranco (research ranking of popular domains), IANA root zone, Public Suffix List, iptoasn.com.