02 - How Large Language Models Work: Tokenization, Embeddings, and Generation

Introduction: Disassembling the Magic of LLMs

Large Language Models seem magical: you type a question and get a coherent, structured, often surprisingly intelligent response. But under the hood there's no magic, there's mathematics. An LLM is fundamentally a system that predicts the next token in a sequence, based on statistical patterns learned from billions of words during training.

Understanding how LLMs work is not an academic exercise: it's an essential practical skill. Knowing what happens between your prompt and the generated response allows you to write better prompts, debug unexpected behaviors, choose the right model for your use case, and understand why LLMs hallucinate.

Phase 1: Tokenization - From Text to Numbers

The first step in LLM processing is tokenization: converting text into a sequence of integers. Neural models don't understand letters or words; they operate on numerical vectors. Tokenization is the bridge between human language and model mathematics.

Byte-Pair Encoding (BPE)

The most common algorithm is Byte-Pair Encoding (BPE), used by GPT, Claude, and most modern models. BPE works iteratively: it starts from individual characters and progressively merges the most frequent pairs into longer tokens.

The result is a vocabulary of 50,000-100,000 tokens representing an optimal compromise between granularity and efficiency. Common words like "the" become a single token, while rare words are split into sub-tokens.

# Example of tokenization with tiktoken (OpenAI's tokenizer)
import tiktoken

# Load GPT-4's tokenizer
enc = tiktoken.encoding_for_model("gpt-4")

# Tokenize a sentence
text = "Generative artificial intelligence is revolutionary"
tokens = enc.encode(text)
print(f"Text: {text}")
print(f"Token IDs: {tokens}")
print(f"Token count: {len(tokens)}")

# Decode each token to see the breakdown
for token_id in tokens:
    print(f"  ID {token_id} -> '{enc.decode([token_id])}'")

# Typical output:
# Text: Generative artificial intelligence is revolutionary
# Token IDs: [Gen, erative, artificial, intelligence, is, revolutionary]
# Token count: 6
      

Practical Impact of Tokenization

Tokenization has important practical consequences every developer should know:

• Cost: APIs charge per token, not per word. One word can be 1-4 tokens
• Context window: the limit is in tokens, not words. 4,000 tokens is roughly 3,000 English words
• Different languages: non-English languages often require more tokens to express the same concept (~1.3x for Italian)
• Code: source code is often less token-efficient than natural text

Phase 2: Embeddings - From Token to Meaning

After tokenization, each token ID is converted into an embedding: a dense vector of real numbers (typically 768-12,288 dimensions) that captures the semantic meaning of the token.

The power of embeddings lies in their geometry: words with similar meanings have nearby vectors in the space. "King" and "Queen" are close, as are "Paris" and "France". These relationships are automatically learned during training.

# Visualizing semantic similarity of embeddings
from openai import OpenAI
import numpy as np

client = OpenAI()

def get_embedding(text: str) -> list:
    """Get the embedding of a text using OpenAI."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def cosine_similarity(a: list, b: list) -> float:
    """Calculate cosine similarity between two vectors."""
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Compare semantic similarities
words = ["cat", "dog", "automobile", "feline"]
embeddings = {w: get_embedding(w) for w in words}

print("Semantic similarities:")
print(f"  cat-feline:      {cosine_similarity(embeddings['cat'], embeddings['feline']):.4f}")
print(f"  cat-dog:         {cosine_similarity(embeddings['cat'], embeddings['dog']):.4f}")
print(f"  cat-automobile:  {cosine_similarity(embeddings['cat'], embeddings['automobile']):.4f}")
# cat-feline will have the highest similarity!
      

Positional Encoding

Beyond semantic embeddings, Transformers add a positional encoding: a signal indicating the position of each token in the sequence. Without this mechanism, the model wouldn't distinguish "the cat chases the mouse" from "the mouse chases the cat", since the Transformer architecture processes all tokens in parallel, not sequentially.

Phase 3: The Transformer - Attention Is All You Need

The heart of every modern LLM is the Transformer architecture, composed of repeated blocks of Self-Attention and Feed-Forward Networks. Models like GPT-4 have hundreds of these stacked blocks, each refining the text representation.

Self-Attention: The Key Mechanism

Self-attention allows each token to "look at" all other tokens in the sequence and decide how relevant they are to its meaning in the current context. In the sentence "The cat sat on the mat because it was tired", the attention mechanism connects "was tired" to "cat" (not "mat"), resolving the coreference.

Mathematically, for each token, three vectors are computed: Query (what am I looking for), Key (what I offer as context), and Value (my informational content). The dot product between Query and Key determines the attention weight, which is used to weigh the Values.

Multi-Head Attention

A single attention mechanism captures one type of relationship. Multi-head attention runs multiple attention operations in parallel (typically 32-128 "heads"), each specialized in a different aspect: syntactic, semantic, proximity, coreference relationships, and so on.

Phase 4: Text Generation

After the input text has passed through all Transformer blocks, the last layer produces an output vector for each position. To generate the next token, this vector is projected onto the entire vocabulary producing logits: a numerical score for every possible token in the vocabulary.

From Logits to Probabilities: Softmax

Logits are transformed into probabilities through the softmax function, which normalizes the scores so they sum to 1. The token with the highest probability is the model's "best prediction", but it's not always the one selected.

Sampling Strategies

The choice of the next token is not deterministic. Different sampling strategies produce outputs with different characteristics:

• Greedy decoding: always picks the most probable token. Deterministic but often repetitive and boring
• Random sampling: samples from the full distribution. Creative but potentially incoherent
• Temperature: controls "randomness". T=0 is greedy, T=1 is the original distribution, T>1 increases creativity
• Top-k sampling: samples only from the k most probable tokens (e.g., k=40)
• Top-p (nucleus) sampling: samples from the smallest set of tokens whose cumulative probability exceeds p (e.g., p=0.9)

# Example: effect of temperature on generation
from anthropic import Anthropic

client = Anthropic()
prompt = "Write the beginning of a fantasy story in one line:"

for temp in [0.0, 0.5, 1.0, 1.5]:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=100,
        temperature=temp,
        messages=[{"role": "user", "content": prompt}]
    )
    print(f"\nTemperature {temp}:")
    print(f"  {response.content[0].text}")

# Temperature 0.0: deterministic output, always the same
# Temperature 0.5: slight variation, still coherent
# Temperature 1.0: creative, good balance
# Temperature 1.5: very creative, possible incoherence
      

Context Window: The Memory of LLMs

The context window is the maximum number of tokens an LLM can process in a single request (input + output). This is effectively the model's "working memory" during a conversation.

Hallucinations: Why LLMs Make Things Up

Hallucinations are one of the most critical problems with LLMs: the model generates false information with the same confidence it generates true information. This happens because an LLM doesn't "know" facts: it predicts the most probable next token given the context.

If the statistical pattern suggests that after "The capital of Australia is" the most likely token is "Sydney", the model will generate "Sydney" even though the correct answer is "Canberra". The model has no internal mechanism to verify the truth of its outputs.

Mitigation: Retrieval-Augmented Generation (RAG)

The most effective strategy to reduce hallucinations is RAG: providing the model with factual information retrieved from reliable sources as part of the context. Instead of asking the model to "remember", we give it updated data and ask it to reason about it.

# Simplified RAG: providing factual context to the model
from anthropic import Anthropic

client = Anthropic()

# Context retrieved from a database or search engine
factual_context = """
Company data updated to Q3 2025:
- Revenue: EUR 12.5M (+23% YoY)
- Employees: 85
- Active clients: 342
- NPS score: 72
"""

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=500,
    messages=[{
        "role": "user",
        "content": f"""Based EXCLUSIVELY on the following data,
answer the question. If the data doesn't contain the answer, say "I don't have this information".

DATA:
{factual_context}

QUESTION: What is the current revenue and how many employees do we have?"""
    }]
)
print(response.content[0].text)
      

Conclusions

Understanding the inner workings of LLMs - from tokenization to generation, through embeddings, attention, and sampling - is not just theoretical knowledge. It's the foundation for using these tools effectively and consciously.

Tokenization influences costs and context limits. Temperature and sampling strategies determine output creativity. The attention mechanism explains why the model understands (or doesn't understand) context. Hallucinations are a direct consequence of the next-token-prediction architecture.

In the next article, we'll put this knowledge into practice with Advanced Prompt Engineering: systematic techniques to get the most from LLMs, from zero-shot to chain-of-thought, from system prompts to the ReAct pattern.