Inference: The Pattern in Motion

Take a copy off the shelf. On my own machine this is literal: a file of a few tens of gigabytes, the quantized seventy-billion-parameter model from the previous essay, loaded into the memory of a graphics card until the card is nearly full. Nothing has happened yet. The structure of learned possibilities is sitting in silicon, fixed, waiting for a question.

I ask one. Say it is about a footbridge I am sketching, how long a timber span can run without a pier. What follows is inference: the pattern in motion. I want to walk through it once in order, because the order explains a great deal of what these systems can and cannot do, and because it is the part of the story that happens a billion times a day.

The prompt goes in

The question is turned into tokens, a few dozen of them. All of them pass through the model at once, every layer, left to right across the stack described in the Transformers essay. At each layer, for each token, the model computes the keys and values that attention will need and keeps them. This store, called the key-value cache, is the model’s working memory for this conversation. It holds the model’s reading of everything said so far, as numbers, and it grows with every token.

At the end of the pass, the representation at the last token is turned into a probability for each entry in the vocabulary. This is the moment the model has been trained for. Somewhere in that distribution is the first token of an answer.

One token at a time

A token is chosen. The simplest way is to take the most likely one. More often one is drawn at random in proportion to the probabilities, with a setting, usually called temperature, that flattens or sharpens the distribution first. Low temperature makes the model predictable; higher makes it varied. The choice is made by the software around the model, and it is the only randomness built into the process.

The chosen token is appended to the prompt, and the cycle repeats. Here the cache earns its keep: the new token is the only thing that needs computing, because the reading of everything before it is already stored. One pass through the stack, one new probability distribution, one more token. On my hardware it is slow enough to watch the sentence assemble. The word-by-word appearance of a chatbot’s reply is not a presentation trick. It is the computation, shown as it happens.

The cycle stops when the model produces a token that means stop, or when a length limit is reached. The answer about the footbridge arrives, and it is useful often enough that I have come to rely on the process. That is my report of my experience, and it is the reason this essay exists.

What moved and what did not

During all of this, the seventy billion parameters did not change. Not one. The question about the bridge produced activations and attention patterns and cache entries, all of which were specific to this prompt and all of which are discarded when the conversation ends. A token changes what the model computes. It does not rewrite what the model is.

This is the single most useful fact for making sense of what a model does and does not remember. The deployed model does not learn from talking to you. Close the window and the bridge question is gone, unless the application chose to store the text somewhere and feed it back next time. Several mechanisms do change a model or extend it, and each deserves its own name so they are not confused with ordinary use:

• Fine-tuning runs the training process again on new data and produces new parameters. It happens offline, in the lab, not during a conversation.
• Adapters add a small set of extra parameters, trained separately, that modify the model’s behavior while the original weights stay frozen. The result is a different system, assembled from parts.
• Retrieval fetches documents from a store the application maintains and places their text into the prompt. The model reads them as context; nothing is learned.
• Application memory is the chat product saving text from earlier sessions and inserting it into later prompts. It lives in a database, not in the model.
• Expert routing, in mixture-of-experts models, chooses which subset of parameters each token runs through. The subsets are fixed; only the choice depends on the input.

Providers may also collect conversations to train future models, under whatever policy they publish. That is a separate act, done later, producing a separate artifact. The model you are talking to is not changing as you talk.

Four sources of an answer

A chat application, as opposed to a bare model, draws on four things when it replies, and keeping them apart tells you how much to trust a given sentence.

The first is the parameters: the cast of the record, the model’s disposition to continue text the way the corpus did. The second is the context: the conversation so far, the system’s instructions, anything pasted in. The third is retrieved documents, placed in the context by the application after a search. The fourth is tool output. A model can be trained to emit a structured request, run this code, look this up, and the application executes it and returns the result into the context. The arithmetic in a good answer about my footbridge may have come from a calculator the model asked for, not from the model’s own disposition to produce plausible numbers.

Only the first of these is “the model.” The other three are the application around it, and the answer you read is a collaboration among all four. When a chatbot cites a document, it was most likely retrieved. When it computes correctly, a tool may have done it. When it states something confidently and wrongly, the parameters were speaking alone.

From one to a billion

Here is where the origin story turns from making a thing to living with it. Training was done once, at great expense, in one place. Inference is done everywhere, cheaply, forever. The same file that I run on one card can be loaded onto thousands of cards in a building, where each card handles many users’ prompts at once, their tokens interleaved so the hardware is never idle. The computation per answer is a small fraction of a cent of electricity. The artifact does not wear out, does not get tired, and does not notice whether it has answered the same question a million times.

This asymmetry is the economic and cultural shape of the technology. A single training run becomes a billion conversations, and the conversations are where the technology actually touches anyone. What limits the spread is not the arithmetic. It is licenses, which for open-weight models range from permissive to conditional and for closed models never release the weights at all, with terms that forbid trying to extract them; hardware, since the file is worthless without a card that can hold it; security, because a weights file that leaks is a model that has escaped its maker’s terms; and the rules governments attach to chips and to the models themselves. Local execution and cloud service are the same pattern in motion, on different scales, under different constraints.

The mystery, located

I have said that a mystery can be accurately described as a mystery. This is the essay where I can say exactly where it sits. Everything above is known: the tokens, the cache, the probabilities, the frozen weights, the four sources. And yet somewhere in the activations that follow my question about a footbridge, a computation occurs that produces an answer I could not have produced as quickly, and sometimes could not have produced at all. We can measure that it works. We can trace, in part, how it works. We cannot yet say, in full, what the structure of learned possibilities is doing when it does. That is a precise statement of an open question, and it is where I want the wonder in this story to live: in the gap between a mechanism we can follow step by step and an outcome we cannot yet fully account for.

The painting

The sixth panel shows the pattern in motion at human scale: a designer at a workstation, her question on the screen beside a half-drawn truss, the answer entering the drawing. The machine is in the room, a single card in an open case, and the racks beyond the glass stand for the same file running at the other scale. Dusk over the city, the Bay Bridge lit. One small diagram on the desk, four arrows into one: parameters, context, documents, tools. No training is happening in this picture. The weights are still. Only the drawing is changing.

The last essay asks who is in the room, and who should be.