Definition
What this term means
The fundamental unit of text that AI models process, roughly equivalent to three-quarters of a word in English. AI systems break all input text into tokens before processing it. Every word, punctuation mark, and space is tokenised. The number of tokens determines how much content fits within a model's context window and how much it costs to process.
Why it matters
The business impact
Token limits affect how much of your content an AI system can consider and process in a single query. Content that is unnecessarily verbose uses more tokens without adding value, potentially causing key information to be truncated or excluded. Writing token-efficient content, concise, clear, and information-dense, increases the likelihood that AI systems will fully process and cite your most important claims.
Used in context
How you might use this term
“During a content optimisation audit, a team discovered that their 3,000-word guide used 4,200 tokens, most of which were repetitive filler. After streamlining to 1,800 words (2,400 tokens) without losing any substantive information, the page's AI citation rate doubled.”
Related terms
Explore connected concepts
Context Window
The maximum amount of text, measured in tokens, that an AI model can process in a single interaction. The context window determines how much information the model can consider when generating a response. Modern models like GPT-4o and Claude support context windows of 128,000+ tokens, but RAG-retrieved snippets are typically much shorter, making concise content crucial for citation.
LLM
A type of artificial intelligence model trained on vast datasets of text to understand, generate, and reason about human language. LLMs power the AI assistants and generative search tools, including ChatGPT, Google Gemini, Claude, and Perplexity, that are rapidly becoming the primary way people discover products, services, and information online.
Embeddings
Dense numerical representations (vectors) that capture the semantic meaning of text. When AI systems convert your content into embeddings, they create mathematical fingerprints that encode what your content is about, its context, and its relationships to other concepts. These vectors are used to measure semantic similarity, enabling AI systems to find content that is conceptually relevant to a query, even if it does not share exact keywords.