Tokens in LLMs: Tokenization, Vocabulary & Why Tokens Aren’t Words

Tokens in LLMs explained simply — what a token is, how tokenization and vocabulary work, why tokens aren’t words, plus token handling in Spring AI.

1. Introduction

Most Spring AI tutorials jump straight to prompts, models and responses. Yet they rarely stop to explain tokens in LLMs. That small gap causes big confusion later. You see a surprise bill, a “context length exceeded” error, or a strange word split, and nothing makes sense.

So in this guide, we will slow down and look at tokens properly. We will learn what a token is, how tokenization works, and why a token is not the same as a word.

This is a foundations article in our Spring AI series. In the earlier articles, we built a ChatClient and connected multiple models. Now we go one level deeper. We open the hood and watch how an LLM actually reads your text.

By the end of this guide, you will know:

• what a token really is, in plain words
• why models use tokens instead of words or characters
• how tokenization turns your text into numbers
• what a model vocabulary is, and how big it gets
• why tokens decide your cost, speed and context limit
• how Spring AI 2.0.1 exposes tokens in real code

Let us start from the very basics.

2. What Is a Token in an LLM?

Let me begin with a simple picture. An LLM cannot read letters or words the way we do. It only understands numbers. So before anything happens, the model chops your text into small pieces. Each piece is a token.

A token is the basic unit of text for a language model. It can be many things:

• a full word, like dog
• part of a word, such as ing, play or tion
• a single character, like a, b or 1
• even a non-English letter, like 你 or á
• a special marker, like an end-of-text signal

So a token is just a chunk. Sometimes it is a whole word. Often it is only a small piece of one.

2.1 Tokens Are Not Words

Here is the part that trips up most beginners. We think in words. The model thinks in tokens. These two do not line up one to one.

Take the word playfulish. It is not in any dictionary. Still, the model handles it with ease. It simply breaks the word into three known pieces:

• play
• ful
• ish

So one “word” became three tokens. This is exactly why the token count is almost always higher than the word count. Because of subword splitting, the model is never stuck on new words, typos or slang. It just builds them from smaller tokens it already knows.

Let me give one more example. The word tokenization often splits into token and ization. So one word becomes two tokens. Spaces usually join the next word too, which adds to the count.

2.2 The Three-Quarter Rule

You do not need exact math every time. A rough rule helps a lot. For English, one token is about three-quarters of a word. So 100 tokens cover roughly 75 words.

Put another way, English text runs at around 1.3 tokens per word on average. Short common words are usually one token. Longer or rarer words split into two, three or more pieces.

Let me show the rule in action. Suppose your prompt holds 400 English words. That is roughly 530 tokens. Now add the model’s 200-word reply. Together you cross 800 tokens in one short exchange. So even a small chat uses more tokens than most people expect.

Keep this rule in your head. It lets you guess prompt size and cost before you even call the model.

3. Why LLMs Use Tokens, Not Words or Characters

A fair question comes up here. Why bother with tokens at all? Why not just use plain words, or plain characters? Both sound simpler. Sadly, both break down at scale. Let me explain why.

3.1 Why Not Just Use Words?

Language is messy, so a word-only system fails fast. Think about these cases:

• Some words are very rare, like unconstitutional.
• Many words are misspelled, like gud instead of good.
• New words appear all the time, like fresh product names.
• A few languages, like Chinese, skip spaces between words.

To cover every possible word, the model would need a giant list. English alone would need 500,000-plus entries, and most would appear rarely. That wastes memory and still fails on anything new.

3.2 Why Not Just Use Characters?

So maybe characters are safer? They are tiny and flexible. However, they create a different problem. Characters are simply too small.

To read unbelievable, the model would process twelve separate steps, one per letter. That makes training slow. It also makes meaning harder to catch. Reading one letter at a time is like reading a book through a keyhole.

3.3 Tokens Are the Sweet Spot

This is where tokens win. A token sits nicely between a word and a character. Common words stay as one token. Rare words split into a few known pieces. Nothing is ever truly unknown.

As a result, the model gets a short and meaningful sequence. It stays efficient, flexible and robust at the same time. That balance is the whole reason tokens exist.

3.4 A Quick Lego Analogy

Here is an analogy that makes it stick. Think of Lego blocks for a moment. Words are like fully built models, rigid and finished. Characters are like single tiny studs, far too small to be useful. Tokens are the mid-size blocks in between.

With a bag of these mid-size blocks, you can build almost anything. A common shape clicks together fast. A rare shape just needs a few more blocks. This is exactly how a model builds any word from its tokens.

4. How Tokenization Works

Now let us trace the actual journey. The process of breaking text into tokens is called tokenization. Behind the scenes, your sentence passes through a clear pipeline.

4.1 From Text to Numbers

The flow has four steps, and the arrows below show the order:

Text → Tokens → Token IDs → Vectors

Here is what happens at each stage:

• First, the tokenizer splits your text into tokens.
• Next, it maps each token to a unique number, called a token ID.
• Then, the model turns each ID into a vector, which carries meaning.
• Finally, those vectors flow into the neural network.

So the model never sees your letters. It works with numbers from start to finish. Take the sentence “I love Spring AI”. The tokenizer may split it into about four or five tokens. Each token then gets its own ID and vector.

4.2 Subword Tokenization

The clever part is subword tokenization. This is what lets a model handle words it has never seen before.

Recall playfulish from earlier. The tokenizer does not panic. It splits the word into play, ful and ish. Each piece already exists in its list of known tokens. Even made-up words become easy this way.

The same trick handles typos and slang. The model simply assembles them from smaller, familiar parts. Thanks to this flexibility, LLMs feel natural with almost any input.

4.3 The Main Tokenization Algorithms

Different model families use different methods. The names sound fancy, but the core idea is the same. Each one learns which pieces are worth keeping as tokens.

• BPE (Byte Pair Encoding): used by the GPT family. It starts from characters and keeps merging the most common pairs.
• WordPiece: used by BERT. It is close to BPE but picks its merges in a slightly different way.
• SentencePiece: used by models like Gemini and Llama. It works smoothly across many languages.

You rarely choose this yourself. The model you pick already decides the algorithm. Even so, it helps to know these names when you read docs or sit in an interview.

4.4 Why One Word Can Split in Different Ways

Here is a subtle point worth knowing. The same word can tokenize differently based on spacing. A leading space often joins the next token. So ” token” and “token” may count as different pieces.

This is why two very similar prompts can have slightly different token counts. You do not control this split directly. The tokenizer decides it, based on how it was trained. Still, knowing this saves you from head-scratching later.

5. Model Vocabulary: The Token Lookup Table

We keep saying “known tokens”. But where does this list live? That brings us to the vocabulary. This is one of the most important ideas in the whole article.

5.1 What Is a Vocabulary?

A model’s vocabulary is the full list of tokens it can recognise. Think of it as a giant table. Each row holds one token and its unique ID.

For example:

• “King” might map to ID 1342.
• “Queen” might map to ID 39.
• “football” might map to ID 8121.

The table stays fixed after training. Every token the model knows lives inside it. When your text arrives, the tokenizer looks up each piece in this table.

5.2 How Big Are These Vocabularies?

Vocabulary size changes from model to model. Bigger is not always better, as we will see soon. Here are some real numbers, checked against trusted sources.

So a GPT-4 model knows about 100,000 tokens. GPT-4o nearly doubled that number. Each jump changes how your text is split, and how many tokens you pay for.

5.3 What If a Word Is Not in the Vocabulary?

This is a great question. And the answer shows why tokens are so smart. Say the vocabulary has apple, phone and table. Now the input word is tabletop. That exact word is missing.

The model still does not fail. Instead, it splits the word into known pieces:

• tabletop → table + top

If top was also missing, the tokenizer would split further, maybe into tab, le, to and p. In the end, every piece exists in the table. The core idea is simple. An unknown word is broken into the smallest number of known tokens.

This is why models never hit a hard wall on new words. There is no “unknown word” error at all. There is only more splitting into smaller pieces.

5.4 Bigger vs Smaller Vocabulary

So why not make the vocabulary huge? There is always a trade-off. A larger vocabulary packs common words into single tokens. That means fewer tokens per sentence, and lower cost per call.

But a big vocabulary needs more memory inside the model. A small vocabulary saves memory, yet it splits text into more pieces. More pieces mean longer sequences and higher cost. So model builders pick a balance that fits their goals.

5.5 A Quick Way to See Tokens Yourself

You do not have to take these numbers on faith. You can watch tokenization happen live. OpenAI offers a free web tokenizer for this. The open-source tiktoken library does the same job in Python.

Paste any sentence and you will see it split into coloured chunks. Try your own name, a long word, or a line of code. Then switch the language to Hindi or Chinese. You will instantly see the token count jump. This small habit builds real intuition about tokens in LLMs.

6. Why Tokens Matter in Real Applications

Fine, tokens are interesting. But why should a Spring developer care? Understanding tokens in LLMs is not just classroom theory. It touches your cost, your limits and your results every single day.

6.1 Tokens Decide Your Bill

LLM providers charge per token, not per word. They count both your input and the model’s output. So a long system prompt costs money on every request.

This adds up quickly in production. Imagine a chat app with thousands of users. A few extra tokens per prompt become a large monthly bill. Therefore, trimming prompts is a real and easy way to save money.

6.2 Tokens Decide the Context Window

Every model has a context window. This is the maximum number of tokens it can handle at once. It covers your prompt, the chat history and the reply together.

Picture a whiteboard with limited space. Once it fills up, old notes get wiped to make room. In the same way, once you cross the token limit, older messages drop off. The model then “forgets” that earlier part of the chat.

6.3 Non-English Text Costs More Tokens

Here is a point many people miss. Tokenizers are trained mostly on English. So English is cheap in tokens. Other languages are often not.

The same sentence can cost far more tokens in another language. With older GPT-4 tokenizers, Hindi could cost several times more than English. Newer tokenizers like o200k_base reduced this gap. Still, if you serve many languages, measure token usage for each one.

6.4 Tokens Also Affect Speed

There is one more effect people often forget. Tokens shape latency too. A model generates text one token at a time. So more output tokens mean a longer wait.

A chatty reply is not only costly. It is also slower for your user. Therefore, a tight output limit helps on both fronts at once. Shorter answers feel snappier and cost less.

7. Working With Tokens in Spring AI

Enough theory. Let us see tokens in real Spring AI code. This ties back to our series nicely. All examples below use Spring AI 2.0.1, the current stable release.

7.1 Capping the Output With maxTokens

Sometimes you want short answers. You can cap the output length with a token limit. Spring AI exposes this through ChatOptions.

var options = OpenAiChatOptions.builder()
        .model("gpt-4o-mini")
        .maxCompletionTokens(300)
        .temperature(0.7)
        .build();
 
String answer = chatClient.prompt()
        .options(options)
        .user(message)
        .call()
        .content();

The maxCompletionTokens value limits how many tokens the model may generate. This keeps replies tight and costs predictable. For generic options across providers, you can also use ChatOptions with maxTokens.

7.2 Tracking How Many Tokens You Used

Guessing token usage is risky. It is far better to measure it. Spring AI returns usage data inside the ChatResponse metadata. For that, use call().chatResponse() instead of call().content().

ChatResponse response = chatClient.prompt()
        .user(message)
        .call()
        .chatResponse();
 
Usage usage = response.getMetadata().getUsage();
Long promptTokens = usage.getPromptTokens();
Long outputTokens = usage.getCompletionTokens();
Long totalTokens  = usage.getTotalTokens();

Now you can log these numbers for each request. Later, you push them to Micrometer and Prometheus. The Spring AI course even wraps this in a reusable TokenUsageAuditAdvisor for exactly this job.

7.3 Splitting Documents by Tokens for RAG

Tokens also matter a lot in RAG. When you load a big document, you cannot send it whole. You split it into chunks first. Spring AI offers a TokenTextSplitter that counts tokens, not characters.

TextSplitter splitter = TokenTextSplitter.builder()
        .withChunkSize(100)
        .withMaxNumChunks(400)
        .build();
 
vectorStore.add(splitter.split(documents));

Here each chunk is about 100 tokens. That keeps every chunk small enough for the model. One nice detail: Spring AI 2.0.1 added stricter validation on the TokenTextSplitter builder arguments. So bad values now fail early, instead of causing silent bugs later.

7.4 Context Window and Chat Memory

Chat memory and tokens are close friends. Memory stores past messages. But the context window limits how many of them actually fit. So you cannot keep everything forever.

By default, Spring AI keeps only the last several messages. For very long chats, that is safer than sending the whole history. For large histories, a vector-store memory picks only the relevant past messages. This way, you respect the token limit and the user still feels remembered.

8. Best Practices and Common Mistakes

Let me share a few simple lessons. These will save you both money and debugging time. None of them are hard, yet they are easy to miss.

8.1 Best Practices

• Keep prompts short and clear. Fewer tokens means lower cost.
• Measure token usage from day one. Never guess your bill.
• Set a sensible maxTokens on replies. It prevents runaway output.
• Count tokens before heavy calls. A tokenizer tool helps a lot here.
• Watch non-English traffic closely. It often costs more tokens.

8.2 Common Mistakes

• Assuming one token equals one word. It almost never does.
• Forgetting that output tokens also cost money.
• Ignoring the context window until you hit a hard error.
• Sending whole documents instead of chunking them first.
• Using the wrong tokenizer to estimate another model’s cost.

9. FAQ’s on tokens in LLMs

10. Conclusion

Tokens sit at the very heart of every LLM. Once you understand tokens in LLMs, a lot of AI stops feeling like magic. You can see why bills grow, why context runs out, and why strange word splits happen.

Here is a quick recap:

• A token is a chunk of text, often just a piece of a word.
• Models use tokens because words and characters both fail at scale.
• Tokenization turns text into tokens, then IDs, then vectors.
• The vocabulary is the fixed list of tokens a model knows.
• Tokens drive your cost, speed and context window.
• Spring AI 2.0.1 lets you cap, track and split by tokens.

Start by counting tokens in your own prompts. Then watch your usage for a week. Small changes here add up to real savings over time.

11. Further Reading

• How LLMs Work: Next-Token Prediction Explained Simply
• ChatOptions in Spring AI: Temperature, Max Tokens & topP Explained
• What Is Spring AI Framework: ChatClient & ChatModel with OpenAI
• Spring AI Reference: Token Usage Handling
• OpenAI Tokenizer