From characters to tokens to numbers — the first step in every LLM pipeline. Interactive explainer with GS Support examples (incl. SEA-language transcripts).
AI models like Claude, GPT, and LLaMA don't understand text the way we do. They work with numbers, not letters. Before any AI can process your message, your text needs to be broken into small pieces called tokens and converted to numbers.
Think of it like this: you're sending a message to someone who only speaks in numbered codes. You need a translation system — that's what a tokenizer does.
You could split text by spaces, but then the AI would need a separate number for every word it's ever seen — including misspellings, slang, and every language on Earth. That's millions of entries. We want a smarter, smaller vocabulary.
Byte Pair Encoding starts with individual characters and repeatedly merges the most common pair into a new token. Over many merges, it discovers useful pieces — common syllables, word roots, suffixes.
Tokenization is the very first step in every LLM pipeline. It determines how the model "sees" your text, affects how many tokens fit in the context window, and directly impacts your cost.
Every API call is billed by token count — both input and output. Understanding tokenization helps you estimate costs and optimize prompts. Numbers and special characters often cost more tokens than you'd expect.
There are four levels at which you can split text. Each has trade-offs:
| Level | How It Splits | Example: "chargebacks" | Trade-off |
|---|---|---|---|
| Word | Each word = 1 token | chargebacks | Simple but huge vocabulary, can't handle unknown words |
| Subword ⭐ | Meaningful pieces | charge + backs | Best balance — what all modern models use |
| Character | Each character = 1 token | charge... | Tiny vocabulary but very long sequences |
| Byte | Raw byte encoding | 9910497114... | Handles any language but extremely long |
Tokenization is Step 1 — everything downstream depends on how text is split into tokens.
BPE is the most widely used tokenization algorithm. It's used by GPT, LLaMA, Claude, and most modern LLMs. The idea is beautifully simple:
Split the entire training corpus into individual characters. This is your starting vocabulary.
Find which two adjacent tokens appear together most often across the corpus.
Combine that pair into a single new token and add it to the vocabulary.
Keep merging until vocabulary reaches desired size or no pair appears more than once.
Let's walk through BPE on the classic textbook example. Watch how common suffixes like "est" and "low" emerge naturally:
low lower newest widest
l o w _ l o w e r _ n e w e s t _ w i d e s t _
Each character becomes its own token. The _ marks end-of-word boundaries.
e + s → esAppears in "newest" and "widest" — 2 occurrences. Merge them.
es + t → estThe suffix "est" emerges! Common in English superlatives.
l + o → loAppears in "low" and "lower" — 2 occurrences.
lo + w → lowThe word "low" becomes a single token! It's frequent enough to earn its own entry.
est + _ → est_The end-of-word suffix "est_" is now a single token.
A common confusion: the BPE playground above shows how the tokenizer is trained — discovering merge rules from a corpus. But when you actually use Claude, the tokenizer works differently:
Happens once, before the LLM is trained. BPE scans a massive corpus (Wikipedia, books, code) and discovers ~50,000-100,000 merge rules. These rules are saved as a file.
Output: merges.txt + vocab.json
This is what our playground simulates ☝️
Happens every time you send a prompt. Your text is split into characters, then the pre-learned merge rules are applied in order. No new rules are discovered — it's a fast lookup.
Input: your prompt
Output: token IDs → embeddings → transformer
"Summarise this IRT case"→ split into characters → apply 100K pre-learned merge rules in order→ ["Sum", "marise", " this", " IRT", " case"] → 5 tokens
Let's see how BPE handles text you'd actually encounter in a D365 case description:
"Pax disputed an SGD $24.50 charge after order cancelled"
A trained BPE tokenizer would split this roughly as:
= 12 tokens. Notice: "Pax" stays one token (common in case data), "SGD" stays one token (currencies are common), but the amount "$24.50" becomes 4 separate tokens — every digit, dot, and currency symbol counts.
BK-2026-4821, SGD 24.50, 2.8%, 2026-03-15 22:14 uses ~25 tokens — mostly because each digit, dash, and punctuation is a separate token. At 400k MIWI cases / month, that adds up.No merges yet — press ▶ to start
Split text into individual characters (+ end-of-word marker ▁)
Count all adjacent token pairs
Merge the most frequent pair into a new token
Repeat until no pair appears more than once
Modern LLMs use one of three subword tokenization algorithms. They all solve the same problem — finding the right-sized pieces — but approach it differently:
| Algorithm | Direction | Merge Criterion | Used By |
|---|---|---|---|
| BPE Byte Pair Encoding |
Bottom-up (merge) | Most frequent pair | GPT LLaMA Claude |
| WordPiece | Bottom-up (merge) | Most likely pair (probability) | BERT |
| Unigram | Top-down (prune) | Remove least useful tokens | T5 |
Start with characters. Repeatedly merge the most frequent adjacent pair. Simple, fast, effective.
Like BPE, but merges the most likely pair based on probability, not raw frequency. Uses ## prefix for continuations.
Start with all possible substrings. Iteratively remove tokens that contribute least to overall likelihood. Opposite direction from BPE.
Every vocabulary includes reserved tokens with specific purposes:
| Token | Purpose | Example |
|---|---|---|
[PAD] | Fills sequences to a fixed length so batches are uniform | case summary [PAD] [PAD] |
[UNK] | Represents any word not found in the vocabulary | the [UNK] rate is |
[CLS] | Class token placed at start (BERT-specific) | [CLS] summarise this case |
[SEP] | Separates two segments | ... data [SEP] assess ... |
[MASK] | Hides a token for the model to predict (BERT training) | the [MASK] rate is high |
Tokens are the currency of AI — literally. Every API call is billed by token count. Understanding tokenization helps you estimate costs, optimize prompts, and choose the right model.
"AnyCompany Support" = 3 tokens
"Case context summary" = 4 tokens
"$24.50" = 4 tokens (each digit, period, dollar sign). Booking IDs, timestamps, and Thai/Vietnamese transcripts cost more tokens than English narrative.
Type text below and watch it transform: characters → tokens → cost. Notice how numbers and special characters create MORE tokens than plain English.
Token estimate uses simplified BPE rules. Actual tokenization varies by model. Pricing as of May 2026 from AWS Bedrock.
| Content Type | Approx. Tokens | Cost (Sonnet 4) |
|---|---|---|
| A short question ("Summarise this case") | ~5 tokens | < $0.001 |
| A 10-line D365 case description | ~150 tokens | $0.0005 |
| Your engineered Case Summarizer prompt template | ~400 tokens | $0.0012 |
| A full case context summary (Symptom · Severity · Booking · Action · Next Step) | ~800 tokens | $0.012 |
| Total per case (input + output) | ~1,350 tokens | $0.013 |
| 10,000 IRT cases per month (one market) | ~13.5M tokens | ~$130 |
| 400,000 MIWI cases per month (region-wide) | ~540M tokens | ~$5,200 |
Simple classification → Nova Micro ($0.04/1M). Narrative generation → Sonnet ($3/1M). Don't use Opus to sort mail.
Remove redundant instructions (10-20% savings). Use shorter examples. Constrain output length.
Bedrock offers 50% discount for batch inference. Perfect for monthly portfolio assessments.
Cache repeated system prompts — saves ~90% on the template portion for subsequent calls.
Tokenization is just the first step. Here's the complete pipeline that turns your text into an AI response:
Your text is split into subword tokens using BPE (or WordPiece/Unigram). Each token maps to an integer ID from the vocabulary.
"case context" → ["case", "context"] → [4523, 8901]
Each token ID is converted into a dense vector of numbers (typically 768–12,288 dimensions). These vectors capture semantic meaning — similar words get similar vectors.
The model needs to know where each token sits in the sequence. Position embeddings are added so "The merchant is risky" and "Is the merchant risky?" produce different representations despite having the same words.
Modern models like LLaMA use RoPE (Rotary Position Embeddings) — which encodes relative position, so the relationship between "merchant" and "risk" is the same whether they're at positions 3,4 or 103,104.
This is where the magic happens. Every token can directly look at every other token and decide which ones are relevant. The model asks three questions for each token:
"What am I looking for?"
"What do I contain?"
"Here's my actual content"
When processing "The merchant's chargeback rate is high", the word "high" pays strong attention to "chargeback rate" (what's high?) and "merchant" (whose rate?). This context-awareness is what makes transformers so powerful.
The transformer produces a probability distribution over the entire vocabulary for the next token. The model picks the most likely token (or samples from the distribution based on temperature), appends it, and repeats.
Input: "The IRT case severity is"
→ P("P1")=0.72, P("P2")=0.21, P("P3")=0.05, ...
→ Output: "P1"
Temperature controls randomness: low temperature (0.0) = always pick the highest probability. High temperature (1.0) = more creative / random. For PAC tagging and severity classification, you want low temperature for consistency. For a coaching note rewrite, slightly higher.
Each breakthrough solved a limitation of the previous approach:
Sparse, no meaning. "cute" and "adorable" are equally different as "cute" and "airplane".
Dense, meaningful, but static — one vector per word regardless of context.
Sequential processing, captures order, but information bottleneck for long sequences.
Context-dependent embeddings via bidirectional LSTMs. "bank" gets different vectors in "river bank" vs "bank account".
Parallel attention over all tokens. No bottleneck. This is what powers Claude, GPT, and LLaMA today.