Token Management Cheat Sheet

Updated 2026-04-28

Next Topic: Transformer Architecture Cheat Sheet

🧠Study flashcards on this topic97 cards · spaced repetition→

Token management is the discipline of controlling, optimizing, and tracking how Large Language Models (LLMs) consume input and output tokens—the fundamental units text is broken into for processing. With tokens driving both API costs (often $1–$30 per million output tokens in 2026) and context window limits (up to 2M tokens), effective token management determines whether AI applications run efficiently or exhaust budgets. Key insight: output tokens cost 3–5× more than input tokens, reasoning models generate thousands of hidden thinking tokens billed at output rates, agentic workflows burn 10–100× more tokens than simple API calls, and even tool schema definitions can silently consume 55K–134K tokens before any work begins—making optimization a production necessity at every layer of your stack.

What This Cheat Sheet Covers

This topic spans 17 focused tables and 117 indexed concepts, 97 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Context Window FundamentalsTable 2: Tokenization AlgorithmsTable 3: Token Counting and EstimationTable 4: Rate Limits and Usage TiersTable 5: Cost Optimization TechniquesTable 6: Context Management StrategiesTable 7: Chunking Strategies for RAGTable 8: Generation ParametersTable 9: Thinking Budget ControlsTable 10: Inference Acceleration TechniquesTable 11: Attention Mechanisms for Long ContextTable 12: Caching and ReuseTable 13: Cost Tracking and ObservabilityTable 14: API Pricing Models (2026 Snapshot)Table 15: Advanced Optimization PatternsTable 16: Streaming and Real-Time ConsiderationsTable 17: Context Extension Techniques

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Context Window Fundamentals

Before you can optimize token usage you have to know what you're actually paying for, and these are the building blocks—input, output, reasoning, and cached tokens—plus the window that caps them all. The asymmetry is the thing to internalize: output costs several times more than input, hidden reasoning tokens bill at output rates, and stuffing the window too full quietly degrades quality long before you hit a hard error.

Concept	Example	Description
Context Window	GPT-5.4: 128K tokens Claude Opus 4.7: 1M tokens Gemini 3.1 Pro: 2M tokens	• The maximum total tokens (input + output) a model can process in a single request • exceeding this causes truncation or errors.
Input Token	User prompt: 500 tokens System instructions: 200 tokens Retrieved documents: 3000 tokens	• Tokens sent to the model (prompts, context, examples) • typically cost $0.50–$5 per million tokens depending on model and tier.
Output Token	Model response: 800 tokens	• Tokens generated by the model • cost 3–5× more than input ($1.50–$30 per million tokens).
Reasoning Token	Claude Opus 4.6 thinking: 8000–32000 hidden tokens o3 reasoning: varies widely per call	• Hidden tokens generated by reasoning models (Claude, o3, Gemini, DeepSeek R1) during internal deliberation • billed at output rates; same prompt can produce vastly different thinking token counts across calls.
Cached Input Token	Reused system prompt: 1200 tokens Cost: $0.30/1M (vs $3.00 uncached)	• Tokens from a reusable prompt prefix stored in KV cache • cost 90% less than fresh input tokens and reduce TTFT latency.
Token Budget	Allocate 70% to history, 20% to retrieval, 10% to response	The planned distribution of available tokens across conversation memory, RAG chunks, tool schemas, and output length.
Context Rot	Model misses earlier instructions at 800K token context depth	• Performance degradation as context fills — the model attends less reliably to older information even when technically within window limits • addressed by compaction and context pruning.
Long-Context Premium	Gemini 3.1 Pro: $2/M input ≤ 200K tokens $4/M input > 200K tokens	• Higher per-token rate charged once a request exceeds a threshold • Gemini reprices the entire request at the higher tier once crossed.
Context Overflow	`Error: Request exceeds token limit`	• When total tokens exceed the context window • model either truncates silently, throws an error, or degrades performance; prevent with budgeting and compaction.

Table 2: Tokenization Algorithms

Tokens don't map cleanly to words—they're produced by subword algorithms that split text into reusable fragments, and which algorithm a model uses determines exactly how your text gets counted and billed. BPE (used by GPT and tiktoken), WordPiece, and SentencePiece each carve up text differently, which is why the same sentence can cost a different number of tokens across providers.

Algorithm	Example	Description
Byte-Pair Encoding (BPE)	`"tokenization"` → `["token", "ization"]`	• Merges most frequent byte pairs iteratively during training • used by GPT models and tiktoken • handles rare words via subword splits.

The rest of this section is part of the full cheat sheet

Table 3: Token Counting and Estimation

When you need a number, you have two choices: a quick heuristic or the model's real tokenizer. The rough ratios here (roughly four English characters per token) are fine for a back-of-envelope budget, but anything touching billing or context-limit validation should use exact, model-specific counting—heuristics break badly on code and non-English text.

Method	Example	Description
Character-to-Token Ratio	English: ~4 characters = 1 token Code: ~2.5 characters = 1 token	• Rough heuristic for estimating tokens without tokenizer access • varies by language (Chinese ~1.5 chars/token, code denser).

The rest of this section is part of the full cheat sheet

Table 4: Rate Limits and Usage Tiers

Providers throttle you on several axes at once—tokens and requests, per minute and per day—and any one of them can stall an app even when the others have headroom. Knowing which ceiling you're hitting (TPM vs RPM vs a daily cap) is the difference between fixing the right problem and randomly retrying; usage tiers then raise those ceilings as your spend grows.

Limit Type	Example	Description
TPM (Tokens Per Minute)	OpenAI Tier 1: 200K TPM Anthropic Standard: 40K TPM	• Total tokens (input + output) allowed per minute • primary throughput control for most providers.

The rest of this section is part of the full cheat sheet

Table 1: Context Window Fundamentals

Concept	Example	Description
Context Window	GPT-5.4: 128K tokens Claude Opus 4.7: 1M tokens Gemini 3.1 Pro: 2M tokens	• The maximum total tokens (input + output) a model can process in a single request • exceeding this causes truncation or errors.
Input Token	User prompt: 500 tokens System instructions: 200 tokens Retrieved documents: 3000 tokens	• Tokens sent to the model (prompts, context, examples) • typically cost $0.50–$5 per million tokens depending on model and tier.
Output Token	Model response: 800 tokens	• Tokens generated by the model • cost 3–5× more than input ($1.50–$30 per million tokens).
Reasoning Token	Claude Opus 4.6 thinking: 8000–32000 hidden tokens o3 reasoning: varies widely per call	• Hidden tokens generated by reasoning models (Claude, o3, Gemini, DeepSeek R1) during internal deliberation • billed at output rates; same prompt can produce vastly different thinking token counts across calls.
Cached Input Token	Reused system prompt: 1200 tokens Cost: $0.30/1M (vs $3.00 uncached)	• Tokens from a reusable prompt prefix stored in KV cache • cost 90% less than fresh input tokens and reduce TTFT latency.
Token Budget	Allocate 70% to history, 20% to retrieval, 10% to response	The planned distribution of available tokens across conversation memory, RAG chunks, tool schemas, and output length.
Context Rot	Model misses earlier instructions at 800K token context depth	• Performance degradation as context fills — the model attends less reliably to older information even when technically within window limits • addressed by compaction and context pruning.
Long-Context Premium	Gemini 3.1 Pro: $2/M input ≤ 200K tokens $4/M input > 200K tokens	• Higher per-token rate charged once a request exceeds a threshold • Gemini reprices the entire request at the higher tier once crossed.
Context Overflow	`Error: Request exceeds token limit`	• When total tokens exceed the context window • model either truncates silently, throws an error, or degrades performance; prevent with budgeting and compaction.

Table 2: Tokenization Algorithms

Algorithm	Example	Description
Byte-Pair Encoding (BPE)	`"tokenization"` → `["token", "ization"]`	• Merges most frequent byte pairs iteratively during training • used by GPT models and tiktoken • handles rare words via subword splits.

Table 3: Token Counting and Estimation

Method	Example	Description
Character-to-Token Ratio	English: ~4 characters = 1 token Code: ~2.5 characters = 1 token	• Rough heuristic for estimating tokens without tokenizer access • varies by language (Chinese ~1.5 chars/token, code denser).

Table 4: Rate Limits and Usage Tiers

Limit Type	Example	Description
TPM (Tokens Per Minute)	OpenAI Tier 1: 200K TPM Anthropic Standard: 40K TPM	• Total tokens (input + output) allowed per minute • primary throughput control for most providers.