Who is crawling your site and why.
Several AI companies now run crawlers that are distinct from Google's main web crawler. They have different purposes and behave differently.
GPTBot is OpenAI's crawler. It collects web content for training data and for ChatGPT's browsing and search features. You can identify it by the user-agent string GPTBot. OpenAI provides a published IP range you can verify against. If your site blocks GPTBot in robots.txt, ChatGPT's live retrieval will not find your content and it may not appear in ChatGPT search results.
PerplexityBot powers Perplexity's answer engine. Unlike models trained on static datasets, Perplexity performs live retrieval on most queries — meaning what your site says today can appear in a Perplexity answer today. This makes PerplexityBot one of the higher-priority crawlers for businesses focused on near-term citation visibility. Google-Extended is the toggle Google added to let site owners control whether their content is used for AI Overviews, Gemini and other Google AI products, separately from organic search indexing. Blocking it prevents your content from influencing those AI surfaces, which is a significant trade-off for most businesses. Additional crawlers include CCBot (Common Crawl, used by many model training datasets), Anthropic's ClaudeBot, and others that will continue to emerge. The general principle applies to all of them: allow the ones that power the surfaces where you want to appear, and make your content easy for them to process.
The technical reality of AI crawling.
Not all content is equal in a crawler's eyes. The gap between what renders in a browser and what a crawler can process is a common source of invisible problems.
Clean HTML with semantic structure.
Most AI crawlers process static or server-rendered HTML reliably. Content inside proper heading tags (H1, H2, H3), paragraph tags and structured lists is parsed with high fidelity. Schema markup (JSON-LD in the document head) is read and processed. Clean URLs, fast page loads and no crawl blocks in robots.txt make the process smooth. This is baseline stuff — but many business sites fail on at least one of these points.
JavaScript-rendered content and visual-only elements.
Content that is only generated by JavaScript at runtime is often invisible to crawlers that do not run a full browser engine. If your key service descriptions, FAQs or case studies are loaded by a JavaScript framework and not present in the initial HTML response, a bot may crawl your page and see almost nothing. Images, PDFs and SVG text are also typically unreadable. Audit your pages with JavaScript disabled to see what a crawler actually encounters.
Answer-shaped passages with clear attribution.
Once a crawler can read your content, the question becomes whether it is worth citing. Crawlers prioritize passages that are self-contained, factual and directly answer a recognizable question. Vague branding copy does not get cited. Direct answers — "TRIVOX AI is a Toronto-based AI automation and GEO agency that builds voice agents, workflow automation and AI visibility systems for GTA businesses" — gives the crawler a citable, attributable sentence. See GEO fundamentals for the content discipline.
Practical steps to improve crawlability for AI bots.
Most of these are not complex. They are often simply not done — which is why fixing them produces noticeable improvements quickly.
Audit your robots.txt file.
Open your robots.txt (yourdomain.com/robots.txt) and check for Disallow rules that might block AI crawlers. A blanket 'Disallow: /' blocks everything. Specific rules blocking directories may inadvertently block your key content. Review and update to explicitly allow GPTBot, PerplexityBot and Google-Extended unless you have a specific reason to block them.
Make your critical content server-rendered.
If your site is built on a JavaScript framework like Next.js, React or Vue, verify that your key service pages, FAQ sections and About content are server-side rendered or statically generated — not loaded exclusively via client-side JavaScript. Use your browser's View Source option to confirm the text is in the HTML response, not in a script block.
Add an llms.txt file.
An llms.txt file at your domain root gives AI crawlers a structured summary of what your site contains and what is most important. Think of it as a sitemap written for language models: clear, plain-text, organized with headings and links. It is a low-effort signal that some crawlers check first before deciding where to go on your site.
Structure your key answer pages with semantic HTML.
Use proper heading hierarchy. Put your most important content in H1 and H2 tags. Wrap your FAQ content in actual question-and-answer elements (even before schema). Use lists where you have lists — do not turn list content into paragraphs to avoid bullet points. Semantic structure helps crawlers parse what is a heading, what is a definition, and what is a supporting detail.
Add JSON-LD schema to every important page.
Organization schema on your homepage tells every crawler exactly who you are, what you do and where you are located. Service schema on your service pages names each offer. FAQ schema on pages with questions makes those Q&A pairs directly available for citation. This is low-cost, high-signal work that directly improves citation frequency.
Related, and what to do next.
Questions, answered.
What are the main AI crawlers I should know about?
The most relevant are GPTBot (OpenAI, used for ChatGPT training and search), PerplexityBot (Perplexity's live retrieval), Google-Extended (Google AI products including AI Overviews and Gemini), CCBot (Common Crawl, feeds many training datasets) and Anthropic's ClaudeBot. Each can be allowed or blocked independently via your robots.txt.
Should I block AI crawlers?
For most service businesses, no. Blocking AI crawlers removes your content from the training data and live retrieval systems that power the AI assistants your potential customers use. There are narrow cases where blocking makes sense — if you are a publisher concerned about training-data use, for example — but for local service businesses wanting to appear in AI answers, allowing crawlers is the right default.
Can AI crawlers read JavaScript-rendered content?
Most cannot, or do so unreliably. Crawlers that do not run a full browser engine see only the initial HTML response. Content that loads via JavaScript after the page renders is often invisible to them. Ensure your key pages are server-rendered or statically generated so the content is in the HTML that a crawler first receives.
What is an llms.txt file?
An llms.txt file is a plain-text document at your domain root that gives AI crawlers a curated, structured summary of your site — what it is, what it contains, and links to your most important pages. It is convention rather than a formal standard, but several AI crawlers check for it. It is a quick, low-effort signal worth adding.
How quickly does being crawled translate to AI citations?
For live-retrieval models like Perplexity, a crawl can translate to a citation within days or weeks. For training-data-based models like the base version of ChatGPT, the timeline is longer and less predictable — training runs happen on their own schedule. Fixing crawlability is worth doing regardless of timeline because it removes an unnecessary barrier to citation.