AI crawler URL inspector

It answers three questions about one URL, and each answer comes from something it measured rather than something it assumed.

• Permission. It fetches /robots.txt from the origin, parses it into user-agent groups under RFC 9309, and reports what that file says about each crawler's access to the site root. The site root — not the path you pasted. That distinction is deliberate and explained below.
• Response. It then makes one real HTTP request per crawler and reports the status code, every redirect hop in order, the content type and the number of bytes returned. This is the half robots.txt cannot tell you: a crawler that is permitted can still be met with a 403, a consent interstitial, a login wall or a redirect to a different page entirely.
• Readability without JavaScript. It takes the HTML that came back, removes <script> and <style> blocks and HTML comments, strips the remaining tags, and counts what is left. That count, against the size of the whole document, is the text share. The bytes inside <script> tags are counted separately as the script share.

This matters enough to state twice. Every request this tool makes carries our user-agent, with the crawler we are asking about named inside it — something of the shape Attensira (asking-as GPTBot; not GPTBot). We do not send GPTBot as our user-agent, because sending somebody's crawler token to a stranger's server is a claim to be that company, and tools in this category make it routinely.

The honest consequence: a site that varies its response by user-agent — a bot-management product, a paywall with a crawler allowlist, an edge rule keyed on the token — may answer the real crawler differently than it answered us. What you get here is a real response from your server to a real request, which is far more than robots.txt gives you, and it is not a recording of what OpenAI or Anthropic saw. We cannot see that, we do not intercept anyone's traffic, and nothing on this page should be read as if we did.

The only record of what a crawler genuinely received is the one your own server wrote. That is a question your access logs answer, not this tool.

The two tools are deliberately not the same check, and running both on the same URL is the point.

• This inspector fetches the page. Its robots.txt column is coarser — it reports the verdict at the site root, because that is the one path every site has — but it adds the thing the other tool cannot see: the status code, the redirect chain and the markup a request actually received.

A page can be allowed by robots.txt and return a 404. It can be allowed and 302 to a login screen. It can return a clean 200 whose body is an empty container waiting for a script. Each of those is invisible to a robots.txt check and each of them ends the same way: nothing to cite.

The two percentages are shares of the same denominator — the bytes the server returned — and they do not add up to 100. The remainder is tags, attributes and CSS: neither readable prose nor executable script.

There is no pass mark here, and we are not going to invent one. No operator publishes a threshold for how much server-rendered text a page needs, so any line we drew would be a number we made up, and it would be the number you acted on. What the split tells you is factual and narrow: an agent that fetches your HTML without running a browser has exactly the words in the text column available to it. If that column is nearly empty on a page you want quoted, the words you want quoted are not in the response.

Whether a given crawler executes JavaScript is a separate question, and not one this tool can answer. Operators change that behaviour without announcing it, and several documented crawlers fetch HTML only. We measure what was served; what each crawler does with it is the operator's to say, which is why every token in the table links to its own operator documentation.

One more limit worth stating: this is a single sample. One fetch, from one place, at one moment. A CDN that varies by region, an origin under load, a deploy mid-flight — any of these change the answer. Treat a single run as evidence to check, not as a settled fact about the page. If a number here matters to you, run it again in an hour.

• Allowed by robots.txt, but the fetch returned 403. Something between the crawler and your origin is refusing on its own terms — a WAF rule, bot management, or a rate limiter that treats unusual user-agents as hostile. robots.txt cannot undo that. The fix is in the same product that imposed it.
• A long redirect chain before the 200. Each hop is a chance to lose the crawler and a chance to lose the canonical signal. Chains built from http to https to www to a trailing slash are the usual cause; collapse them into one hop.
• Blocked at the root, but the page itself matters. A blanket Disallow: / is usually a staging file that shipped. Write the file you meant with the robots.txt generator and re-run both checks.
• 200, HTML, and almost no text. The page renders client-side. Server-rendering the primary content is the real fix; a summary in the HTML is a partial one. Once the words are in the response, check how they are organised with the content structure analyzer.
• An empty mount point in the markup with no text around it. The inspector reports this only when it actually parsed an element with no children — it is a fact about the bytes, not a guess about your framework. It usually travels with a very low text share, and it has the same fix.