What Claude Is Actually Doing: LLMs Without the Magic
I spend a lot of my day talking to large language models. I build tools around them, I pipe them into my terminal, I let them draft code and answer questions. And the single most common thing I hear from people who don’t work with them daily is some version of the same question:
Okay, but what is it actually doing when it answers me?
There are two wrong answers that both feel right. The first is magic: it just kind of understands, somehow, don’t ask. The second is lookup: it’s basically a very fancy search engine that finds the answer in a giant database. Both are comforting, and both are wrong. The gap between them and the truth is where all the confusing behaviour lives, and it’s why the same model can be brilliant one moment and confidently make something up the next.
So let me try to close that gap. No math degree required. But I’m not going to hand-wave the statistics either, because the statistics are the point. By the end you’ll have poked at a handful of little models-in-miniature running right here in your browser, and watched the whole thing run as one loop, so that “what is it actually doing” feels a lot less mysterious.
First surprise: it doesn’t read letters, and it doesn’t really read words
Before a model can predict anything, your text has to be turned into something it can do math on. That step is called tokenization, and it’s weirder than you’d expect. The model doesn’t see letters. It mostly doesn’t even see words. It sees tokens: chunks of text, each mapped to a plain number.
Have a look. Pick a few of these and watch how they get chopped up:
Pick some text. This is roughly how the model chops it into tokens — the actual units it reads. Each token is just a number.
3 tokens. The famous one. The model sees three chunks, not "s-t-r-a-w-b-e-r-r-y". That makes counting letters harder — though research suggests the bigger issue is that models are just bad at counting repeated things in general, tokenization or not.
␣ marks a leading space that belongs to the token. IDs are illustrative, not from a specific tokenizer.
Notice strawberry? To you it’s a word with three r’s. To the model it’s three chunks: str, aw, berry, and it never sees the individual letters directly. For years this was the go-to explanation for why LLMs botch “how many r’s are in strawberry,” and it’s part of the story. Asking a model to reason about letters inside a token is a bit like asking you to count the pixels in a photo you’re looking at. But recent research has walked this back as the explanation. Controlled studies find models can often identify the individual letters in a word just fine, and still miscount them. The sharper bottleneck seems to be counting itself: LLMs are generally bad at tallying repeated things, tokenization or not. Tokenization makes it harder. It isn’t the whole reason.
Everything after this point, every bit of “reasoning,” every clever answer, is the model working with these numbered tokens. So keep that in mind: it’s all just numbers going in and numbers coming out.
And that raises the question everyone forgets to ask. Numbers come out, so how do they turn back into words on your screen? The answer is refreshingly boring. That token-to-number mapping in the demo above runs in reverse, via a lookup table, no model required. Token ID 15717 always means berry, full stop. The vocabulary is a fixed, static list baked in alongside the model, the same one used to tokenize the input in the first place. So “detokenizing” isn’t a skill the model has to learn or reason about at all. It’s just: take the ID the model picked, look up its text in the table, glue it onto the output, repeat. All of the intelligence is in picking the right token. Turning it back into text is a one-line dictionary lookup.
The one trick: predict the next token
Here’s the part that sounds too simple to be true. A model like Claude has exactly one core skill: given the tokens so far, predict the next token.
That’s it. That’s the whole engine. It reads everything up to now, your question, its own answer as it writes it, and produces a probability for every possible next token. Not one guess. A full ranked list, with a number attached to each option, all adding up to 100%.
Modern Claude does more on top of this. It can think through a problem in a visible “thinking” phase before answering, or call tools and read the results mid-conversation. But zoom into any single moment of any of that, and it’s still this exact loop producing the next token. The extra structure is built on the trick, not instead of it.
Play with it. This little model is trying to continue “The cat sat on the”:
For the next token, the model isn't looking anything up. It produces a probability for every possible token. Here are its top guesses:
␣mat ␣floor. ␣couch. ␣roof. ␣windowsill. ␣keyboard. Low temperature sharpens the bars toward the single best token (reliable, repetitive). High temperature flattens them (surprising, sometimes nonsense). Same model, same knowledge — only the dice change.
A few things I want you to actually feel here, not just read.
Those percentages are real predictions, not confidence in a fact. When the model says mat is 61% likely, it isn’t saying “I’m 61% sure the cat sat on a mat.” It’s saying “across the mountains of text I learned from, sentences that start this way continue with mat about this often.” It’s a bet shaped by patterns, not a fact retrieved from a shelf. There is no shelf.
Generation is just this, over and over. Hit “Sample next token” a few times. Each click, the model rolls a weighted die over its list, appends the winner, and then predicts again from the new, slightly-longer text. A whole paragraph is nothing more than this loop running hundreds of times. There’s no plan for the sentence, no outline it’s filling in. Just the next token, and the next, and the next. Real systems usually trim the die first, cutting off the long tail of implausible tokens before rolling, but the core idea, weighted chance over a ranked list, is exactly this.
Now drag the temperature slider. This is my favourite part, because temperature doesn’t give the model new knowledge. The underlying bets are identical. It only changes how sharply it prefers its top guess. Turn it down and the bars spike; the model becomes predictable, repetitive, “safe.” Turn it up and the bars flatten; low-probability tokens get a real shot, and the output gets creative, then eventually incoherent. When people say an AI is being “more creative” or “more random,” this dial is usually what’s moving. Same model. Same knowledge. Different dice.
Try “Always take the top pick” too. That’s temperature zero, pure greedy determinism. Reliable, and a little boring: exactly the trade-off.
How does it know which words matter?
If a model only ever predicts the next token, how does it handle sentences where the answer depends on something ten words back? Consider: “The trophy didn’t fit in the suitcase because it was too big.” What’s “it”? You know instantly it’s the trophy. But nothing in grammar tells you that. You know because a big thing failing to fit is what makes sense.
Models solve this with a mechanism called attention. When processing a word, the model looks back over the earlier words and decides, for each one, how much it matters right now. It’s not a rule. It’s a set of learned weights.
Hover over the highlighted words to see where each one “looks”:
To predict the next token well, the model has to figure out which earlier words matter. This "looking back" is called attention. Hover or tap a highlighted word below to see what it leans on.
Looking at it,
the strongest connection is to trophy. That's how the model resolves what "it" refers to — not with a grammar rule, but
by weighting the words that best explain the sentence.
Flip the adjective and watch where it points. One
word rewires the whole interpretation — no rule anywhere says so, it's all learned weights.
Focus on it. See how strongly it connects back to trophy? That connection is the model resolving the ambiguity, and it learned to do that purely from seeing how language behaves, never from a grammar textbook. Now hit the button to swap “big” for “small.” One adjective changes, and it now points at the suitcase instead, because now it’s the container that’s the problem. No hard-coded rule could catch every case like that. The weights can, because they were shaped by enormous amounts of real text.
Stack a lot of these attention layers together, feed tokens through them, and you get a transformer, the architecture under basically every modern LLM. Anthropic doesn’t publish Claude’s exact internals, and most 2026 frontier models bolt on efficiency tricks: routing tokens through only part of the network, sharing attention machinery across heads. But attention doing the “looking back” is the common thread through all of them. You don’t need the wiring diagram. The intuition is enough: look back, weigh what matters, predict the next token.
But a sentence is easy. How does a whole essay stay coherent?
This is the objection I always get. Fine, it can figure out “it” in one sentence, but how does it write three coherent paragraphs, keep a character’s name straight, not contradict a decision it made 400 words ago? If it’s only ever predicting the next word, shouldn’t it drift into nonsense?
The answer is that attention doesn’t only reach back a few words. It reaches back across the entire text so far, and it does so at different scales at once. When the model writes a new word, it can lean on a specific earlier word, on a whole earlier sentence, and on an entire earlier paragraph, all at the same time. That multi-scale “looking back” is what holds a long output together.
Watch it happen. Here’s a little story. Pick a word the model is about to write and see how far back, and at what scale, it reaches to stay consistent:
Coherency over a long text isn't magic either. As the model writes each new word, it can attend far back — to specific words, to whole sentences, and to entire paragraphs. Pick a word it's about to write and see what it reaches back to:
Mara had always been the careful one. On the morning she left home, she packed a single brass key and told no one where she was going.
At the crossroads outside town, she had to choose a direction. The southern road was faster, but she decided to head north, toward the mountains.
Three days later, cold and tired, she reached the locked gate.
…writing next: "Mara"
To name the character consistently, it reaches all the way back to paragraph 1, where "Mara" was introduced — not to a nearby word.
That’s the whole trick behind coherency. There’s no separate “memory” module and no outline being followed. Every single token is generated by looking back over everything written so far and weighting what still matters: the name from paragraph one, the direction chosen in paragraph two. So the next word fits not just the last few words, but the whole story. Stretch that across thousands of tokens and you get something that reads like it was planned, even though it was only ever predicted one token at a time.
This is also why a model has a context window: a limit on how far back it can look in a single pass. Current frontier models are generous about this. Some of Claude’s models support up to a million tokens, roughly a small library shelf of text. Everything inside the window can be attended to directly. What falls outside it isn’t necessarily gone forever, though. Production systems increasingly compress or summarize older conversation rather than dropping it outright, so “forgotten” is more like “no longer in full resolution” than “erased.”
So where do all these probabilities come from?
This is the step people most often imagine as magic, so let me be concrete. During training, the model is shown staggering amounts of text and plays one game, billions of times. The text is cut off at some point, the model guesses the next token, and it’s corrected against what actually came next. Guess well, barely adjust. Guess badly, nudge the internal numbers so it’d do better. Repeat until the nudging settles.
Here’s that game, slowed down to one lap. Watch the bar for the true word climb across passes as the same pattern gets reinforced:
A snippet from the training data is cut off before the next word:
The sun rose over the?
Pass 1 over this handful of examples. Watch the ` horizon` bar in step 2 — across passes it climbs, because this exact pattern (and millions like it) has been reinforced before. Real training runs this loop across the entire internet, essentially, not four sentences.
What comes out the other side is mostly not a database of facts. It’s billions of tuned numbers, called parameters, that together encode the statistical shape of language: which words tend to follow which, how ideas connect, what a valid sentence feels like. The “knowledge” is baked into those weights the way the shape of a landscape is baked into a river’s path. That’s why a model can usually answer questions it never saw verbatim, and mostly can’t quote its sources. There are no sources filed away in there, just compressed patterns.
So how does “Paris is the capital of France” end up in there without a row in a database somewhere? Most of that heavy lifting happens in a part of the network sitting right next to the attention layers, called the feed-forward or MLP layer. Interpretability researchers who’ve dug into these layers describe them as working like a giant key-value lookup: one stage of the layer recognizes a pattern in what it’s currently processing (something like “this text is asking about a country’s capital”), and the next stage nudges the output toward the associated answer. It’s not a clean row-and-column table. Facts, grammar rules, and vague associations all get packed into the same few thousand numbers per layer, overlapping and interfering with each other, a phenomenon researchers call superposition. There are far more things worth knowing than the model has room to store cleanly, so it compresses, trading a bit of precision on any one fact for room to hold far more of them.
“Mostly” is doing real work in that “mostly compressed patterns” sentence a couple paragraphs up, though. Researchers have shown large models can occasionally memorize and spit back near-verbatim chunks of specific training text, especially anything unusual or repeated often enough. It’s a documented exception to the “just patterns” picture, not just a theoretical one.
That compression is also what reframes a lot of behaviour that otherwise seems random. A pattern that shows up constantly in training, grammar, common code, standard explanations, gets a strong, well-reinforced spot in that shared space, which is why the model is great at it. A fact that shows up once, an obscure date, a niche API, gets a much fainter, more crowded one, and it’s easier for the “nudge” to land slightly wrong, or on a plausible neighbor instead of the exact answer.
Why it can be confidently wrong
Now the hallucination question finally has a clean answer. The model’s job, its only job, is to produce a plausible next token. Not a true one. A plausible one. Usually plausible and true line up, because true things are what mostly appear in the training text. But when the pattern is thin, the model still does its job. It emits the most probable-looking continuation, and probable-looking is not the same as correct.
Go back to the temperature demo and picture a question where the real answer barely registers: a citation that doesn’t exist, a function that was never written. The bars over the wrong-but-plausible options are tall. The correct token’s bar is a sliver, if it’s there at all. The model samples confidently from a distribution that’s simply pointed the wrong way. It isn’t lying and it isn’t broken. It’s doing exactly what it always does, in a spot where “plausible” and “true” came apart. Understanding that is the difference between being surprised by a confident mistake and expecting one where the facts are thin.
That’s the mechanical root of it, but it’s not the only reason confident wrongness sticks around. The training and evaluation that happens after pretraining matters too. If a model is graded, by people, by benchmarks, or by its own later training, in a way that rewards a fluent, confident-sounding guess more than an honest “I don’t know,” it will learn to guess fluently and confidently. Thin knowledge plus an incentive to never shrug is a bad combination. That’s part of why “just tell it to say when it’s unsure” helps less than you’d hope.
Putting it all together
So far you’ve seen each piece on its own. But the whole point is that they’re one loop, turning constantly. Every token Claude produces is a full lap through the same four steps:
- Tokenize: give the new token its number. (The text before it was already numbered on earlier laps.)
- Attend: look back over all of it and weigh what matters right now.
- Predict: produce a probability for every possible next token.
- Sample and append: pick one, stick it on the end, and go back to step 1.
Here’s that loop running by itself. Hit play and watch a passage build itself, one lap at a time:
The starting context becomes tokens:
The detective
Token 1 of 10. The passage is scripted so it stays coherent while it plays; the point is the shape of the loop, not the exact words. Real systems don't redo the tokenize and attend steps over the whole growing text each lap. They cache the earlier tokens' processed state (the "KV cache") and only do fresh work for the newest token, which is a big part of why generation is fast.
That’s it. That’s the entire machine. No step is doing anything mysterious. The intelligence is in how well each prediction is made, multiplied across thousands of laps. Watching it run, I think the surprising thing isn’t that it sometimes gets things wrong. It’s that a loop this simple, run this many times, produces anything coherent at all.
One thing the demo simplifies: real systems don’t redo the tokenize and attend steps over the whole growing text on every lap. Once a token has been processed, the model caches its internal state (called the KV cache) instead of recomputing it, so each new lap only does fresh work for the newest token. That caching is a big part of why generation is fast, not just how it stays correct.
So, what is Claude?
Everything above is the whole machine: tokens in, look back and weigh what matters, predict a probability for every next token, sample one, repeat. Claude is that same loop, scaled up enormously, and then aligned. After the raw prediction training, Anthropic tunes it further against a written set of principles, Claude’s Constitution in their terms, the successor to their earlier Constitutional AI approach, so its behavior reflects being helpful, honest, and safe. That’s why it behaves like an assistant instead of an autocomplete that trails off. But underneath the polish, the engine is the one you just played with.
In production, that loop doesn’t run on its own. It’s wrapped in classifiers that screen the request on the way in and the response on the way out, a separate check from the model’s own training, built to catch what shouldn’t get through in either direction:
A normal request
Can you explain what a hash map is?
Nothing flagged — a completely ordinary request.
tokenize → attend → predict → sample
Generates the answer normally, token by token.
Nothing flagged — streams straight through.
—
There isn't one rule for how a refusal happens. Sometimes nothing is flagged and the model declines on its own — that's trained behavior, not a classifier. Sometimes a classifier blocks a request before the model ever runs, and the reply you see was never generated at all. And sometimes the model starts answering for real and a classifier watching the stream cuts it off partway through. Illustrative pipeline, grounded in Anthropic's published Constitutional Classifiers work and Claude API docs — not a full description of the production internals.
There’s no little person in there, no encyclopedia, no lookup table. There’s a probability distribution over the next token, shaped by an incomprehensible amount of language, running in a loop, and out of that falls something you can hold a conversation with. It’s statistics, at a scale that starts to look like magic from the outside. Now you’ve seen the inside.
The demos above use hand-authored, illustrative numbers so they run entirely in your browser, no live model, no network calls. The behaviour they show is accurate. The exact token IDs, attention weights, and percentages are made up to keep things clear.