AI crawler checker: can ChatGPT and Google's AI Overviews read your website?

Robots.txt is a short text file at the root of your website, at yourwebsite.com/robots.txt, that tells crawlers which pages they may read. Search engines and AI companies send crawlers to read websites, and their answers draw on what those crawlers could read. Google, for example, says (opens in a new tab) a page has to be indexed and eligible for a snippet in Google Search to appear as a link in its AI Overviews. In our test of AI answers in six towns, ChatGPT linked senior living communities’ and operators’ own websites 12 times when asked about memory care.

Not out of ChatGPT search. OpenAI runs separate crawlers: GPTBot collects content that may be used to train its models, and OAI-SearchBot finds websites for ChatGPT’s search answers. OpenAI says (opens in a new tab) sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers”, though they may still appear as plain navigational links, and that a robots.txt change can take about 24 hours to reach its search results. A site can block GPTBot and still allow OAI-SearchBot.

Google says (opens in a new tab) Google-Extended “does not impact a site’s inclusion in Google Search”. It controls whether content Google crawls may be used to train Gemini models and to ground answers in the Gemini apps. For AI Overviews and AI Mode, the control is robots.txt for Googlebot, the same crawler as for search. Google also plans to support a shared signal for AI training preferences in early 2027; it isn’t live yet, so this check doesn’t read it.

Yes. A firewall, a security plugin or your hosting company can refuse a crawler whatever robots.txt says. Some check where a request comes from, and the real crawlers come from their owners’ published addresses, so they can be treated differently from our test. Our test request comes from our own server, so it can’t show what your firewall does with them. Your firewall’s or host’s logs show it, and for Googlebot, the URL Inspection tool in Google Search Console fetches a page as Google does.

They fetch a page when a person asks a chat assistant about it, rather than crawling the web. OpenAI says (opens in a new tab) robots.txt rules “may not apply” to ChatGPT-User and points site owners to OAI-SearchBot for search opt-outs, and Perplexity says (opens in a new tab) its user fetcher “generally ignores robots.txt rules”, so a robots.txt result for those two would mislead. Anthropic says (opens in a new tab) Claude-User follows robots.txt.

From a robots.txt file someone edited, an SEO or security plugin that writes rules for you, your hosting company’s bot protection, or a content delivery network’s settings. If your site uses Cloudflare, its settings for AI crawlers are under Security, then Settings, filtered by Bot traffic, in “AI Crawler and Agent access policies (opens in a new tab)“, with separate choices for search, agent and training crawlers. Cloudflare’s documentation, updated November 2, 2026, lists “Allow” as the default for each. Check the setting on your own account rather than assume the default.

They are Anthropic’s crawlers. ClaudeBot collects content that could be used to train Anthropic’s AI models; Anthropic says (opens in a new tab) blocking it “signals that the site’s future materials should be excluded” from its training data. Claude-SearchBot reads websites for Claude’s search results, and blocking it keeps your content out of that search index. Both follow robots.txt.

Not necessarily. Access is the first condition: an answer can only draw on pages its crawler may read. Whether it names your business depends on what your website, your Business Profile, your reviews and your listings say about you. Our AI answer visibility page explains that work, and our free online presence check scores how AI answers show your business today. For senior living operators, our free audit does this for each community.

No. We keep a count of checks and their results by month, not the websites typed in. A check’s result is kept for one hour, so the same site isn’t fetched twice in a row, and then it expires. We don’t ask for your email or your name.