Token
LLM & Language ModelsThe basic unit of text that AI models process — roughly equivalent to a word or word fragment. AI pricing, context windows, and rate limits are all measured in tokens.
Tokens are the atoms of AI language processing. Before an AI model reads your text, it breaks it into tokens using a tokenizer. In English, one token is roughly 3/4 of a word — a common short word is usually a single token, a long or unusual one splits into several pieces, and a 1,000-word essay comes to about 1,300 tokens. Exact counts depend on which tokenizer the model uses.
Tokens matter for three practical reasons: (1) pricing — model APIs bill per token, quoted per million, with input and output usually priced differently, (2) context window — a 128K token limit works out to roughly 96,000 words of total conversation, and (3) generation speed — models produce tokens one at a time, so longer responses take longer.
Understanding tokens helps you optimize AI usage: shorter prompts cost less, staying within context limits prevents conversation amnesia, and knowing your token budget helps you plan complex workflows.
Real-World Example
Model pricing and context limits are quoted in tokens, never in words. Knowing the rough conversion — about 0.75 words per token — is what lets you turn "a 200K-token context window" into "roughly a 150,000-word document" before you commit to an approach.
Related Terms
Try AI Rewriter
Rewrite and improve any text while preserving meaning and adding a human touch.
Try FreePut this concept to work
Once the definition is clear, the next useful move is to try a focused tool flow instead of bouncing through more glossary pages.
Open the rewriter routeFAQ
What is Token?
The basic unit of text that AI models process — roughly equivalent to a word or word fragment. AI pricing, context windows, and rate limits are all measured in tokens.
How is Token used in practice?
Model pricing and context limits are quoted in tokens, never in words. Knowing the rough conversion — about 0.75 words per token — is what lets you turn "a 200K-token context window" into "roughly a 150,000-word document" before you commit to an approach.
What concepts are related to Token?
Key related concepts include Context Window, LLM (Large Language Model), API (Application Programming Interface), Rate Limit, Tokenization. Understanding these together gives a more complete picture of how Token fits into the AI landscape.