# What Is Tokenization, and Why Does It Decide What AI Costs?

Original: https://urielpapa.com/blog/what-is-tokenization
Author: Uriel Papa
Topic: AI, in Depth
Published: 2026-10-02

> Every AI model reads, writes, limits and bills you in tokens, not words. Once you understand tokens, AI pricing and limits start to make sense.

## Key takeaways

- Tokenization is how a large language model splits text into small pieces called tokens, which can be whole words, parts of words or punctuation.
- Different models use different tokenizers, so the same sentence can become a different number of tokens in Claude, GPT or Gemini.
- Model providers charge per token, usually with separate prices for input tokens and output tokens.
- A model's context window is measured in tokens, so tokenization also decides how much text a model can work with at once.
- What surprised me on real bills was how much of the cost came from the system prompt and chat history sent with every message, not from the question itself.

Tokenization is how a language model splits text into small pieces called tokens; models read, write, limit and charge by tokens, not words. That is why two prompts of the same length in words can cost different amounts, and why a long document can suddenly be "too long" for a model.

This explainer is part of my [AI, in Depth](/topics/ai) writing. I use models through OpenRouter in my own automations, so token counts are something I watch on real bills. I first noticed token costs when I built a Messenger and Instagram chatbot with n8n and OpenRouter, where every customer message also carried the full system prompt.

## What is a token?

A token is the basic unit a large language model works with. It can be a whole word, part of a word, a single character, a space attached to a word, or a punctuation mark. Before a model sees your text, a tokenizer turns it into a list of tokens, and each token into a number the model can process.

Models do this because working with whole words would need an enormous vocabulary and would fail on new words, names and typos. Working with single characters would make every text very long. Tokens sit in between: common words become one token, and rare words get split into familiar pieces.

## How does text become tokens?

Here is an illustrative example. The exact split depends on the tokenizer, and every model family uses its own, so treat this as a picture of the idea rather than the output of a specific model.

Sentence: "Tokenization isn't magic, it's math."

| #  | Token    | Note                                                 |
| -- | -------- | ---------------------------------------------------- |
| 1  | Token    | A common word piece                                  |
| 2  | ization  | A common suffix                                      |
| 3  | " isn"   | Many tokenizers attach the leading space to the word |
| 4  | 't       | Contractions are often split                         |
| 5  | " magic" | A common word becomes one token                      |
| 6  | ,        | Punctuation is usually its own token                 |
| 7  | " it"    |                                                      |
| 8  | 's       |                                                      |
| 9  | " math"  |                                                      |
| 10 | .        |                                                      |

So a 5-word sentence becomes about 10 tokens in this example. Short sentences with contractions and punctuation split more than average. For ordinary English prose, [OpenAI's rule of thumb](https://developers.openai.com/api/docs/concepts) is that one token is about 4 characters, or 0.75 words, so 1,000 words come to roughly 1,300 tokens. Most providers offer a tokenizer tool or a token-counting API, and that is the reliable way to count tokens for a specific model.

<Callout type="note">
  Tokenizers differ between models. The same text can be a different number of tokens in Claude, GPT, Gemini or Gemma, so always count with the tokenizer of the model you plan to use.
</Callout>

## Why do tokens decide what AI costs?

Providers charge per token, usually quoted per million tokens, with one price for input (what you send) and a higher price for output (what the model writes back). Every request is billed as input tokens times the input price plus output tokens times the output price.

These are the standard API prices for one mid-range model from each of the three big providers, checked on October 17, 2026:

| Model                               | Input per 1M tokens | Output per 1M tokens | Cached input per 1M tokens | Source                                                                        |
| ----------------------------------- | ------------------- | -------------------- | -------------------------- | ----------------------------------------------------------------------------- |
| Claude Sonnet 5.5                   | US$2.00             | US$10.00             | US$0.10                    | [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing) |
| OpenAI gpt-6.1-sol                  | US$2.00             | US$10.00             | US$0.10                    | [OpenAI pricing](https://developers.openai.com/api/docs/pricing)              |
| Gemini 3.1 Pro Preview (up to 200K) | US$2.00             | US$12.00             | US$0.20, plus storage      | [Gemini API pricing](https://ai.google.dev/gemini-api/docs/pricing)           |

Output costs five to six times as much as input on all three, which is why long replies drive most of a bill. Prices change often, so check the source pages before you budget.

The input side includes more than your question. A chatbot request often carries a system prompt, the conversation so far and any documents you added, and all of that is counted every time.

## Why do tokens set a model's limits?

A model's context window is the maximum number of tokens it can consider in one request, counting both what you send and what it writes back. When a conversation or document goes past that limit, something has to be dropped or summarized. That is why long chats can seem to "forget" what was said early on.

Output length is limited in tokens too. A model may stop mid-answer when it reaches its output limit, even if the context window still has room.

## What does this mean in practice?

In practice, tokens turn a vague AI cost into a number you can estimate. Here is a rough example from the Messenger and Instagram chatbot I built with n8n and OpenRouter for my former agency, VITE SEO. In a typical month it handled around 1,000 messages. Each request carried about 2,000 input tokens, mostly the system prompt and recent chat history, and each reply was about 150 tokens. That comes to around 2 million input tokens and 150,000 output tokens a month.

On a small, low-cost model at about US$0.15 per million input tokens and US$0.60 per million output tokens, that is well under US$1 a month. At the mid-range prices in the table above, it would be about US$5 to US$6. These are estimates based on the usage OpenRouter shows in its activity page, not an exact bill.

After looking at the numbers, I cut the system prompt to about half its length, sent only the last few messages instead of the whole chat, and asked the model for short replies. The biggest saving came from the system prompt, because it was sent with every single message.

To estimate the cost of an AI feature before you build it:

1. Write a typical request: the system prompt, a sample user message and any context you would include.
2. Count its tokens with the tokenizer for your chosen model.
3. Estimate the length of a typical reply in tokens.
4. Multiply by the input and output prices, then by the number of requests you expect each month.
5. Add a margin for long conversations and retries.

## How can you use fewer tokens?

* **Keep the system prompt short.** It is sent with every request, so every extra sentence costs you on every message.
* **Trim the conversation history.** Send the last few messages or a short summary instead of the full chat.
* **Ask for short, structured replies.** Output tokens usually cost the most.
* **Pick the right model size.** A smaller model can handle simple tasks like classification for a fraction of the cost.
* **Use caching where the provider offers it.** Most major providers discount repeated input, such as a long system prompt, when it is sent again within a short time. As of October 2026:

  * **OpenAI** caches prompts of 1,024 tokens or more automatically, and cached input is [discounted by up to 95 percent](https://developers.openai.com/api/docs/guides/prompt-caching) depending on the model.
  * **Anthropic** caches what you mark with `cache_control`. A cache read costs [0.1 times the normal input price](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) or less, writing to the cache costs a little more than normal input, and the cache lasts 5 minutes by default, with a paid 1-hour option.
  * **Google Gemini** applies [implicit caching](https://ai.google.dev/gemini-api/docs/caching) by default on Gemini 2.5 and newer models and passes the savings on automatically, with explicit caching available for more control.

  Rates change often, so check each provider's pricing page before you budget around them.

Context windows deserve a post of their own, and they are one of the topics in this hub.

## How this affects business

- AI bills vary because token counts vary with language, formatting and how long the replies are.
- Estimating tokens per task before building gives a realistic monthly cost for an AI feature.
- Output tokens usually cost more than input tokens, so short, structured replies save money.

## Questions people ask

### How many words is a token?

It depends on the tokenizer and the language. OpenAI's rule of thumb is that one token is about 4 characters, or about three quarters of a word, in English text, so 100 tokens is roughly 75 words. Other languages, code and unusual words often take more tokens for the same meaning.

### Why do output tokens cost more than input tokens?

Most providers price output higher because the model generates output one token at a time, which takes more compute than reading the input. Check each provider's pricing page for the actual rates.

### Do local models use tokens too?

Yes. A local model running in a tool like LM Studio still tokenizes text and still has a context window measured in tokens. You pay in hardware and electricity instead of per-token fees.