A single robots.txt line can silence your AI visibility.
Most business websites were set up before AI crawlers existed. Their robots.txt files were written to block bad actors and thin scrapers — and they often inadvertently block good bots in the process.
A common configuration is a wildcard block: User-agent: * / Disallow: /admin — which seems harmless but can cascade depending on how it is written. More problematic: some SEO tools and WordPress plugins add aggressive crawl-blocking that excludes user-agents they do not recognize. Since GPTBot and PerplexityBot are relatively new, older security configurations often block them by default. The result is that your site is perfectly indexed in traditional search but completely invisible to AI retrieval systems.
The fix is straightforward, but it requires actually opening and reading your robots.txt. This is low-effort and often overlooked. We see it consistently when running AI visibility audits for GTA businesses: a clean site, strong content, solid GBP — and a robots.txt that blocks every bot except Googlebot. The entire AI layer of visibility is off by default, not by choice.
Which AI bots to know by name.
Each major AI product has its own crawler user-agent. You address them individually in robots.txt when you want to give specific instructions.
GPTBot — ChatGPT's crawler.
GPTBot collects content for OpenAI's training datasets and for ChatGPT's live search and browsing features. Allowing GPTBot means your content can appear in ChatGPT search results and may be included in future training updates. OpenAI publishes its IP ranges publicly so you can verify crawls are legitimate. Blocking GPTBot in robots.txt is the primary reason businesses do not appear in ChatGPT answers for direct queries.
PerplexityBot — live retrieval.
Perplexity performs live web retrieval on most queries, which means PerplexityBot is one of the more time-sensitive crawlers for citation purposes. Content that PerplexityBot can access today can appear in a Perplexity answer today. Blocking it — intentionally or by accident — removes your site from one of the fastest-growing AI search surfaces. For businesses focused on near-term AI citation visibility, allowing PerplexityBot is a priority. See how AI crawlers read your site for the full picture.
Google-Extended — AI Overviews and Gemini.
Google-Extended is the user-agent that controls whether your content is used for Google's AI products: AI Overviews (which appear at the top of search results for many queries) and Gemini. This is separate from Googlebot, which controls organic search indexing. You can block Google-Extended while still being indexed in standard search. For most businesses, allowing Google-Extended is the right call — it is how you appear in AI Overviews without any additional effort beyond having good content.
How to write a robots.txt that works for AI visibility.
Here is the practical approach, written for someone who may not be a developer.
Find and read your current robots.txt.
Navigate to yourdomain.com/robots.txt in your browser. If nothing appears, the file may not exist — which means all crawlers are allowed by default, and you can skip to step three. If the file exists, read it carefully. Look for 'Disallow: /' under any User-agent (especially a wildcard User-agent: *), which would block everything.
Check for blanket blocks.
The most common problem is a wildcard block that was intended to limit scrapers but blocks all unknown bots including AI crawlers. A line like 'User-agent: * / Disallow: /' blocks everything. Look also for plugins or security tools that add aggressive Disallow rules — common in WordPress, Shopify and similar platforms.
Explicitly allow the citation crawlers.
The safest approach for a service business is to add explicit Allow entries for GPTBot, PerplexityBot, Google-Extended, ClaudeBot and CCBot. You can do this with a simple block at the top of your robots.txt for each user-agent: 'User-agent: GPTBot / Allow: /' followed by the same pattern for the others. This overrides any later blanket blocks for those specific bots.
Block only true scrapers with no AI benefit.
There are crawlers that harvest contact data, resell scraped content or exist purely for competitive intelligence with no AI citation benefit. These are worth blocking. Examples include AhrefsBot and SemrushBot for certain content types, and various known spam crawlers. This is a separate category from AI citation bots — do not conflate them.
Test after every change.
After editing your robots.txt, use Google Search Console's robots.txt tester (or any free robots.txt validator) to confirm your intended bots are allowed on your most important pages. Check specifically for your homepage, your main service pages and your FAQ pages. Re-audit every time a plugin update or platform change might have modified the file.
When blocking AI crawlers makes sense.
For completeness: there are legitimate reasons to block some AI crawlers. They are narrower than most people assume.
If you are a publisher whose business model depends on content licensing and you are concerned about your content being used in AI training without compensation, blocking GPTBot and CCBot is a reasonable position. Several media companies have taken this stance. If you have genuinely proprietary data — methodologies, research, tools — that you do not want ingested into training corpora, blocking certain crawlers makes sense for that content specifically, even if not site-wide.
For service businesses, clinics, agencies and local operators in the GTA, the calculus is different. Your content is marketing material. The goal is for as many people — and AI systems — as possible to know what you do, where you are and why you are credible. Blocking AI crawlers from a service business site is, in almost all cases, a mistake. The only exception: if you have a private portal, a client area or proprietary backend content, blocking those specific paths makes sense, and you should already have them behind authentication anyway.
Related, and what to do next.
Questions, answered.
What is robots.txt and how does it affect AI crawlers?
robots.txt is a plain text file at the root of your website that tells crawlers which parts of your site they can access. AI crawlers like GPTBot and PerplexityBot respect this file. If your robots.txt blocks them — intentionally or by accident — your content cannot be retrieved or cited by those AI systems.
Should I block GPTBot?
For most service businesses, no. Blocking GPTBot prevents your content from appearing in ChatGPT search results and removes it from consideration for OpenAI training data. Unless you have a specific concern about your proprietary content being used in AI training, allowing GPTBot is the right default for a business that wants AI visibility.
What is Google-Extended and should I allow it?
Google-Extended is the user-agent Google uses for AI products including AI Overviews and Gemini — separate from the standard Googlebot that powers organic search. Blocking Google-Extended means your content will not appear in AI Overviews even if it ranks well in standard search. For most businesses, allowing it is the better choice.
How do I check if my robots.txt is blocking AI crawlers?
Navigate to yourdomain.com/robots.txt in your browser and read the file. Look for Disallow rules under User-agent: * (which applies to all bots) or specifically under the user-agent names of AI crawlers. Use Google Search Console's robots.txt tester or a free robots.txt validator to confirm which bots can access which pages.
If I allow AI crawlers, will they crawl my entire site?
They will crawl pages they can discover via your sitemap or internal links, within their own crawl rate limits. You can allow them site-wide while still blocking specific sensitive directories — for example, allowing all of your public pages while blocking /admin and /client-portal. This is the recommended approach for most businesses.
Does fixing my robots.txt immediately improve AI visibility?
For live-retrieval models like Perplexity, yes — the effect can be visible within days once the bot can actually reach and index your content. For training-data models, the timeline is longer and less predictable. But fixing a blocking robots.txt is the single fastest structural change that can affect AI citation visibility, because it removes a complete barrier.