AI crawler log audit New / beta

New / beta. This feature has not been run on real customer logs yet: it was tested on synthetic logs only, built from the nginx, Apache and Cloudflare Logpush format documentation.

Which AI crawlers really visit your site, which ones are impostors using a fake User-Agent, and which of your pages they never fetch. Checked against the IP lists the crawler operators publish.

1. Filter your log here, in your browser

Accepted: nginx or Apache "combined" logs (also with an X-Forwarded-For field), Apache "common", Cloudflare Logpush HTTP requests JSON. .gz is fine. Up to 5 MB and 50,000 lines after filtering.

Choose an access log

2. Send it to the API ($0.50 per audit)

POST https://aeoinspector.com/api/v1/log-audit?pages=/,/pricing with the file as the body, paid per call in USDC on Base with an x402 wallet or agent. Card checkout is not available yet. Invalid, unrecognisable or oversized logs get a 400/413 before payment and are never charged.

Prefer the command line? This keeps only crawler lines:

grep -aiE 'Google-CloudVertexBot|meta-externalfetcher|meta-externalagent|Claude-SearchBot|Perplexity-User|MistralAI-Index|meta-webindexer|MistralAI-User|Amzn-SearchBot|Kimi-SearchBot|OAI-SearchBot|PerplexityBot|DuckAssistBot|ChatGPT-User|Claude-User|GoogleOther|DuckDuckBot|OAI-AdsBot|Bytespider|ClaudeBot|Googlebot|Amazonbot|Amzn-User|Kimi-User|cohere-ai|Applebot|HySpider|bingbot|KimiBot|GPTBot|CCBot' access.log | gzip > ai-crawlers.log.gz

Privacy

• Filter first (recommended): this page reads your log in your browser, throws away every line that does not name a known crawler (your human visitors), and removes query strings, referrers, cookies and user names from the rest. Nothing leaves your device until you send the filtered file yourself.
• If you send a log to the API, our server does receive it. It is processed in memory while the request runs and then discarded: we do not write it to storage, logs, analytics or usage counters, and the response holds only aggregates: counts per crawler, the paths crawlers requested, and, for requests that pretend to be a crawler, a coarse /24 (IPv4) or /48 (IPv6) network instead of the address.
• Our host, Cloudflare, records request metadata for our service (for example time, URL, status and the caller's IP address and user agent), and Workers Logs are turned on. Our code does not write the log body anywhere; Cloudflare's own logging follows Cloudflare's policies.

Privacy policy for the log audit (beta)

What you send. The log audit (this page and the API at POST /api/v1/log-audit) analyses a web server access log that you choose to send us. Access logs can contain personal data about your own visitors, such as IP addresses, user agents, page addresses with query strings, and referrers. You decide what to send, and you are responsible for having the right to share it with us.

Filtering in your browser (recommended). This page reads your file inside your browser. Before anything leaves your device it removes every line whose user agent does not name a known crawler (your human visitors), removes query strings from page addresses, and keeps only these fields for crawler lines: time, method, path, status code, user agent, the requesting IP address and the first X-Forwarded-For address. It also lists up to 20 of your most-requested page paths (paths only, no addresses) for the coverage check. Nothing is uploaded until you send the filtered file yourself. The page loads no analytics or third-party scripts.

What our server does with a log you send. Our server does receive what you send. It processes the log in memory while your request runs and discards it when the response is returned. We do not write the log, or any line or address from it, to a database, file, key-value store, our application logs, analytics or usage counters. While the log is processed we read only the time, method, path (query string removed), status code, user agent and client IP address of each line; other fields such as referrers, cookies and user names are ignored. Each IP address is used only to check whether it is inside the IP list published by the crawler operator the user agent names.

What we send back. The response contains aggregates: request counts per crawler, status code counts, crawler-requested paths that returned errors, the time window covered, and which of the paths you listed were never fetched by an AI crawler. For requests that claim to be a crawler but come from outside that operator's published IP list, we return the request path and a shortened network (a /24 for IPv4 or a /48 for IPv6), never the full address. Visitor addresses are never returned.

Records about the call itself (not your log). Paid calls use the x402 protocol. When a payment settles we record the payer wallet address, amount, asset, network, transaction hash, route and response status in our payment database and in one line of our Workers logs; this is the same public data that is on the blockchain. When a call is answered with 402 Payment Required we record the time, the site host name, route (without query string), method, the caller's user agent and referrer, the caller's country and network operator (ASN), and whether a payment header was present; we do not record the caller's IP address or the request body. We also keep anonymous counters of settled and unsettled calls.

Our hosting provider. The service runs on Cloudflare Workers, and Workers Logs are turned on for it. Cloudflare records request metadata for each call, for example the time, the URL called (including any pages= list you put in the URL), the response status, and the caller's IP address and user agent, under Cloudflare's own policies (https://www.cloudflare.com/privacypolicy/). Our code does not write the request body to those logs.

Retention. Log content: none beyond the request. Records about the call and platform metadata: as described above.

Terms for the log audit (beta)

1. Your data and your right to send it. You may send only logs you are entitled to share. If they contain personal data about your visitors, you are responsible for having a lawful basis to share it with us for this analysis and for telling your visitors where your own policies require it. Use the in-browser filter to send only crawler lines whenever you can.
2. Limits. One request may contain up to 5 MB (measured after gzip decompression) and up to 50,000 lines. Lines longer than 8,192 characters are skipped and counted. Requests over a limit, and requests in which no line can be read as an access log line or Cloudflare Logpush record, are rejected with an error before payment and are not charged.
3. Price and payment. The audit costs the price shown in our API description at the time of the call (currently $0.50 per audit), paid in USDC via x402. Requests rejected for invalid or oversized input are not charged. A charge is settled only when an audit result is returned.
4. What the result means. A user agent is only a claim. "In vendor ranges" means the request IP was inside the IP list the operator published when we last fetched it; "outside vendor ranges" usually means impersonation but can also mean the operator changed its list since. Some operators publish no list, so their requests cannot be confirmed either way. Results describe the log you sent and are provided as is, without a guarantee of completeness or accuracy. This is a beta feature.
5. No storage. We do not keep your log, so we cannot re-send or reproduce a result later. Save the response if you need it.

Crawlers recognised (31): GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, MistralAI-Index, MistralAI-User, Googlebot, GoogleOther, Google-CloudVertexBot, bingbot, Applebot, DuckDuckBot, DuckAssistBot, Amazonbot, Amzn-SearchBot, Amzn-User, meta-externalagent, meta-externalfetcher, meta-webindexer, CCBot, KimiBot, Kimi-SearchBot, Kimi-User, HySpider, Bytespider, cohere-ai. Privacy policy · Terms