Why ChatGPT can't see your site

The four usual reasons ChatGPT never mentions or links your site: robots.txt, a firewall or CDN block, content that only appears after JavaScript runs, and broken URLs. How to check each one.

You ask ChatGPT about your category and it names three competitors, but never you. Sometimes that's about content and reputation. Often, though, it's something more boring: ChatGPT's crawlers can't get to your pages, or they get there and find nothing to read. Those problems are fixable, and you can check for them in a few minutes.

First, know which OpenAI bot does what

OpenAI runs several crawlers, and each one is controlled separately (OpenAI's crawler documentation):

OAI-SearchBot finds pages for ChatGPT's search features. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links.
GPTBot collects content that may be used to train OpenAI's models. Blocking it is a training opt-out. It is not what decides whether you show up in ChatGPT search.
ChatGPT-User visits a page when a person's request in ChatGPT triggers it. OpenAI notes that robots.txt rules may not apply to these user-initiated visits, and that ChatGPT-User isn't used to decide what appears in search.

So if you care about being found in ChatGPT, the bot that matters most is OAI-SearchBot. Plenty of sites block GPTBot on purpose and that's fine. Blocking OAI-SearchBot by accident is the expensive mistake.

1. robots.txt says no

Open yoursite.com/robots.txt. Look for a group that names OAI-SearchBot, and for a catch-all User-agent: * group with Disallow: /, which blocks every crawler that doesn't have its own group. Staging-site rules that slipped into production are a common cause.

To allow ChatGPT search while opting out of training, you can write:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

OpenAI says it can take about 24 hours after a robots.txt change for its search systems to adjust, so don't expect an instant result.

2. A firewall or CDN blocks the request

robots.txt is only a request. Your CDN or firewall can still refuse the crawler before it ever reads the file. On Cloudflare, AI bot policies are split into three behaviors, Search, Agent and Training, and each can be set to block on all pages, block only on pages with ads, or allow (Cloudflare: Block AI Bots). Cloudflare also documents new defaults for new domains from September 15, 2026: bots classified as Training or Agent are blocked on pages that display ads, while Search stays allowed.

The catch is that a block here overrides whatever robots.txt says. If your robots.txt welcomes OAI-SearchBot but a firewall rule returns 403 to it, the result is the same as a block. OpenAI also publishes the IP ranges its crawlers use (linked from the same OpenAI page) and recommends allowing them. That matters if your firewall allowlists by IP.

3. Your content only exists after JavaScript runs

A study of crawler traffic by Vercel and MERJ found that OpenAI's crawlers (OAI-SearchBot, ChatGPT-User and GPTBot) fetched JavaScript files but did not execute them, so content rendered only in the browser wasn't visible to them (The rise of the AI crawler, December 2024). Crawler behavior can change, but if your homepage arrives as an empty <div id="root"> that gets filled in later, assume an AI crawler sees the empty version.

A quick test: view the page source (not the inspector) and search for a sentence from your main copy. If it isn't in the raw HTML, server-side rendering or pre-rendering is the fix, at least for your key pages.

4. The URLs it tries are broken

The same study found that ChatGPT's crawler spent about 35% of its fetches on 404 pages and another 14% following redirects. Old URLs, redirect chains and stale sitemaps waste the visits you do get. Keep your XML sitemap current, redirect old URLs in a single hop, and make sure your canonical URLs return 200.

What fixing these won't do

Being reachable is the entry ticket, not a guarantee. Once OAI-SearchBot can read your pages, whether ChatGPT mentions you depends on what's on them and what the rest of the web says about you. But if the crawler can't get in, none of that has a chance to count.

Check it in 30 seconds

The free check below reads your robots.txt rules for 21 AI crawlers, including OAI-SearchBot, GPTBot and ChatGPT-User, fetches your homepage the way those crawlers identify themselves to see whether a firewall answers them differently, and counts the words that are visible without JavaScript.

1 free check a day per person (per browser) · works without JavaScript · no signup