Loading…
Color-coded token boundaries with counts — see what models see.
Each colored block ≈ one token. Whitespace shown as ␣ merges into the following token — roughly how BPE tokenizers behave.
🎨 This is a close visual approximation (words ≤5 chars = one token, longer words split at 4 characters, CJK ≈ 1 token/char), not the exact tokenizer of any specific model — use the GPT Token Counter for cost math. Great for building intuition about why some phrases cost more.
LLMs see chunks, not words: a long word may cost three tokens while a short one costs one. Watching your text split builds the intuition that makes prompts cheaper and truncation surprises rarer.
Words up to 5 characters form one chunk, longer words split at 4-character boundaries, CJK characters count individually, and whitespace merges into the following chunk — mirroring BPE behavior closely enough for intuition.
Exact tokenization varies per model and tokenizer version. For cost math use the GPT Token Counter; use this to understand structure: why code eats tokens, why line breaks matter, why translations surprise.
Each color block is one approximate token; spaces render as ␣ and attach to the next chunk. Long unbroken strings (URLs, hashes, base64) visibly explode into many chunks — exactly why they bloat prompts.
The Token Visualizer handles token visualizerdirectly in your browser. Paste or type your input, and the tool processes it instantly — no upload, no signup, no waiting. It's built for the moments when you need a quick transformation and don't want to leave your workflow.
Because the tool runs client-side, it's fast and private. Your text never touches a server, which makes it safe for sensitive content. The interface is keyboard-friendly and works on any device with a modern browser.
Common uses: people reach for this tool when they need to use a how does tokenization split words, visualize gpt tokens online, why does my text cost so many tokens, or token boundaries color highlight.
Browser-based tools like this one have a few real advantages over installed software or manual methods:
The Token Visualizer is based on the following formula:
tokens ≈ characters ÷ 4 words ≈ tokens × 0.75 cost = tokens ÷ 1,000,000 × price per 1M tokens
Variables: tokens = Approximate token count shown by the visualization characters = Character count of the text 1 token ≈ 4 characters of typical English prose common word = Usually 1 token; rare or long words split into several subword tokens price per 1M tokens = Model rate ($ per 1,000,000 tokens) cost = Estimated cost to process the text once ($)
Models read subword pieces, not whole words: frequent short words like 'the' are a single token, while rare or long words break into several, so token count follows character patterns rather than word count. The color-coded boundaries make each piece visible, which is why the ≈ 4 characters per token rule is only an average.
Worked example: Step 1: a 600-character English paragraph → tokens ≈ 600 ÷ 4 = 150. Step 2: sanity check: 150 tokens ≈ 150 × 0.75 ≈ 112 words. Step 3: a rare word like antidisestablishmentarianism (28 characters) can cost 5+ tokens while the costs 1 — that is why character count is only an estimate. Step 4: at $3 per 1M input tokens: cost = 150 ÷ 1,000,000 × 3 = $0.00045. Result: sending the paragraph once costs about $0.00045, and 10,000 such calls would cost about $4.50.
More tools you might find useful