Glossary

AI Crawler

Automated bots that index web content for AI training or retrieval systems.

Definition

What this term means

Automated bots that index web content for AI training or retrieval systems.

The business impact

Allowing AI crawlers access ensures your content can be discovered and cited.

How you might use this term

“We configure robots.txt to explicitly allow GPTBot, Claude-Web, and other AI crawlers.”

Explore connected concepts

Robots.txt

A file that guides crawlers on which pages to access.

ai.txt

An emerging standard for declaring AI usage and access preferences.

Common Crawl

A nonprofit that maintains a free, open repository of web crawl data used by many AI systems.

Ready to improve AI visibility?

Understanding the language of AI visibility is the first step. See how your brand performs across AI systems with a free scan.