Which AI crawlers actually visited your site
Everything else written about AI crawlers is a prediction. Your access log is the only record of what really happened: which crawlers requested your pages, on which days, and what your server answered them. Drop a piece of that log in below and you get the breakdown — verified visits, impostors using a crawler's name from an address that isn't theirs, and the pages where a crawler was handed a 403 instead of your content. The file is read inside this page and is never uploaded anywhere.
The file is read inside this page. It is not uploaded, not stored and not sent anywhere — you can check that in your browser's network tab.
Supported: nginx and Apache (combined and common), Caddy JSON, log-shipper exports where the raw line sits in a JSON field (Datadog, Loki, Fluent Bit, docker logs), and JSON logs from Vercel and Cloudflare. Gzipped files work as they are.
Why the log settles arguments that robots.txt cannot
A robots.txt file states a policy. It says who is allowed in. It says nothing about who showed up, how often, or what they received when they did. Plenty of sites allow every AI crawler and still never get visited, and plenty of sites believe they are open while a firewall rule, a bot-protection service or a stale redirect quietly refuses every request. Both cases look identical from the outside and completely different in the log.
The single most useful thing this tool surfaces is not the visit count — it is the responses that were not 200. A crawler that was refused took your page out of the running for the questions your buyers ask, and it did so silently, weeks before anyone noticed. That line in your log is worth more than any score, because it names a specific page, a specific status code, and a specific date you can go and fix.
How a claimed crawler is separated from a real one
A User-Agent header is a string the client chooses. Anyone can send "GPTBot" and many people do — scrapers, security scanners and competing audit tools all wear other people's names because it gets them past naive filters. Counting user agents alone inflates the numbers, sometimes by half. So every request here is checked against the IP ranges the operators publish themselves: Google, OpenAI, Anthropic, Perplexity and Apple all list theirs. An address inside the list is a verified visit. An address outside it, for an operator that publishes a list, is an impostor.
Two more corrections happen quietly. A single address wearing three or more different crawler names is not nine robots visiting — it is somebody's audit tool, and its requests are subtracted rather than counted. And requests for CSS, JavaScript, fonts and images are separated from requests for pages, because sixty hits from a crawler can easily mean one page and fifty-nine assets. Both adjustments are shown in the report rather than applied invisibly.
What this tool is not
- It is not a live check. It reads history that your server already recorded, so it can only tell you about the period your log file covers.
- It is not proof that a crawler read or used your content. A successful fetch means the page was delivered, nothing more — no provider publishes what happens after that.
- It is not a verdict on operators that publish no IP ranges. Those visits are marked unverified, never labelled fake, because there is nothing honest to check them against.
Questions about reading crawler logs
- Where do I find my access log?
- On a server you control, look in /var/log/nginx/access.log, /var/log/apache2/access.log or /var/log/caddy/access.log. On a managed platform, look for log export or log drains in the dashboard. A gzipped file works as is — there is no need to unpack it first.
- Which log formats are supported?
- The combined and common formats from nginx and Apache, Caddy's JSON lines, log-shipper exports that wrap the raw line in a JSON field (Datadog, Loki, Fluent Bit, docker logs), and JSON logs from Vercel and Cloudflare. Gzipped files are read as they are. IIS W3C logs are not supported yet. If your format is not understood you get a plain refusal rather than an empty report, and the shape of a failing line is sent so the parser can learn it.
- Does my log file get uploaded?
- No. The file is read and analysed inside this page, and it never leaves your browser. If our parser fails to understand your format, an anonymous shape of the failing line is sent so we can fix it — letters become a, digits become 9, so no address, path or domain survives. The full lines are only sent if you press the button and see exactly what would go.
- Why is a crawler I know visits me shown as unverified?
- Because its operator does not publish IP ranges, so there is nothing to check the address against. Meta, ByteDance and Amazon are in this group today. Unverified means we cannot confirm the visit, not that we think it is fake.