How Large Language Models Work
A first-principles guide to transformer attention, next-token prediction, sampling mathematics, and why AI models hallucinate.
Pieces in this series
How Large Language Models Generate Text
From raw prompt strings to token embeddings, self-attention calculations, and next-token probability sampling
Large language models do not think, reason, or look up answers in a hidden database. They are statistical machines that predict the single most probable next token in a sequence, repeating that calculation billions of times per minute.
Why AI Chatbots Sometimes Make Things Up
The mechanics of hallucination, probability vs. truth, and the engineering behind grounded AI responses
AI hallucinations are not software bugs or accidental glitches. They are the direct, natural consequence of training a neural network to maximize statistical plausibility rather than factual truth.
How Transformers Changed AI
From sequential recurrence and vanishing gradients to scaled dot-product attention, multi-head projections, and modern neural scaling laws
Before 2017, natural language processing relied on recurrent neural networks that processed text word by word, bottlenecking GPU parallelization and forgetting long-range context. The Transformer threw away recurrence entirely. By…
How Mixture of Experts (MoE) Scales Large Language Models
From dense transformer bottlenecks and sparse gating to top-k routing, expert capacity, and Mixtral parameter scaling
Dense language models activate every parameter for every single token, coupling compute costs to model capacity. Sparse Mixture of Experts (MoE) architectures decouple parameter count from inference compute. By replacing monolithic…
In the scriptorium — this series continues to unfold