GEO glossary
The bots with which model providers read the web, for training or for real-time search. They must be recognised and allowed.
AI crawlers are the programs with which model providers read the web. Some collect data for training (OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, Common Crawl's CCBot), others search in real time to answer a question (OAI-SearchBot and ChatGPT-User, PerplexityBot and Perplexity-User). They identify themselves in robots.txt by name.
Many hosts, CDNs and firewalls block them by default, together with generic anti-bot rules. For a company that wants to be cited it is an own goal: an unreadable website cannot become a source. Checking crawler access (robots.txt, firewall rules, server logs) is among the first technical checks of every assessment.
Free audit in 48 hours
A free demo audit, no commitment: real AI answers about your brand and an indicative GEO Score, presented on a call.