---
title: "What Is a Token in a Large Language Model?"
slug: what-is-a-token-in-an-llm
category: ai
category_label: "AI"
author: "BrainWavePost Staff"
date: 2026-05-30
tags: ["tokens", "LLM", "tokenization", "OpenAI"]
read_time_minutes: 6
canonical_url: https://brainwavepost.com/article/what-is-a-token-in-an-llm
source: BrainWavePost
---

# What Is a Token in a Large Language Model?

*AI · 2026-05-30 · BrainWavePost Staff · 6 min read*

> A plain-English explainer on what a 'token' is when working with large language models, with citations to the official documentation of OpenAI and Google.

> **How this article is sourced** _(info)_
>
> Definitions and rules of thumb come from OpenAI's Help Center and Google Cloud's Vertex AI documentation. [1][2]

When you send text to a large language model (LLM), the model does not read characters or words directly. It reads tokens — short chunks of text that the model has been trained to recognise. [1]

## What counts as a token?

OpenAI's Help Center explains that 'tokens can be thought of as pieces of words.' For English text, OpenAI gives the following rough rules: 1 token ≈ 4 characters, 1 token ≈ ¾ of a word, and 100 tokens ≈ 75 words. [1]

Google's Vertex AI documentation describes the same concept for Gemini models, noting that a token can be a single character, part of a word, or a whole word, and that exact counts depend on the tokenizer used by the specific model. [2]

## Why tokens matter

Tokens determine two practical things: the context window (the maximum amount of text the model can read at once) and the cost (most LLM APIs are priced per million tokens of input and output). [1][2]

## How to count tokens

OpenAI publishes an open-source tokenizer called tiktoken that returns the exact token count for a given string and model. The free online 'Tokenizer' tool on the OpenAI platform site does the same in a browser. [1][3]

> **Quick rule of thumb** _(tip)_
>
> For English: a typical page of plain text (~500 words) is roughly 650–700 tokens. Code, non-Latin scripts and emoji often use more tokens per character. [1]

## References (clickable)

- [1] OpenAI Help Center — 'What are tokens and how to count them?': https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
- [2] Google Cloud — Vertex AI documentation on tokens for Gemini models: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/tokens
- [3] OpenAI Platform — Tokenizer tool: https://platform.openai.com/tokenizer

---

_Canonical article: [https://brainwavepost.com/article/what-is-a-token-in-an-llm](https://brainwavepost.com/article/what-is-a-token-in-an-llm) — © BrainWavePost. Educational content; see the article page for full disclaimers._