Web scraping
Which AI crawlers do major sites actually block?
46 sites, 18 AI agents, read from robots.txt. ClaudeBot was blocked by 22 and GPTBot by 16, and the ordering is strict.
Research
Measurements we ran ourselves, published with the method, the versions and the date.
Everything in this section is a first-party measurement. Where a study reports a number, the page states how it was produced, on which software versions, and on what date, and prints enough of the method that you can run it yourself.
That last point is the one that matters. The purpose is not that you should trust our result; it is that you should be able to get your own. Client behaviour changes between releases, so a measurement is a fact about a version on a day rather than a permanent property, and each study says so explicitly.
We publish no provider performance tables. No independent party has counted the pools, and reprinting a vendor figure as though it were a measurement would be worse than printing nothing.
Anti-bot systems
Browsers vary every TLS handshake. Libraries never do.
Twelve connections from one Chromium browser produced twelve different JA3 hashes. curl and Go produced one each, every time.
Anti-bot systems
What HTTP clients send before you configure anything
Nine HTTP clients and a browser, captured at the byte level. The browser sent 11 headers; the median library sent 4, in a different…