Training: From Sand to a Learned System

Take the metaphor literally for a moment. Sand is mostly quartz, and quartz is silicon bonded to oxygen. Heat it with carbon and the oxygen leaves. Purify what remains until the unwanted impurities are measured in parts per billion. Melt it, draw it into a single crystal the size of a fire hydrant, slice the crystal into wafers thinner than a credit card, and polish them until they reflect like water. Then print on them. The printing is done with light, in machines built by one company in the Netherlands, in factories concentrated in Taiwan, and it lays down transistors by the tens of billions on a piece of silicon you can hold between two fingers.

That chip does one thing well. It multiplies large grids of numbers by other large grids of numbers, very fast, many at once. Put a few thousand of them in a building with enough power and cooling, connect them with cables fast enough that they can share results in microseconds, and you have the machine on which a model is trained. None of this is metaphor. It is mining, chemistry, optics and plumbing, and it happens in specific places that can be visited.

The previous essay ended with a corpus: a selected slice of the human record, broken into tokens. This one follows those tokens into the machine and explains what happens there. The short version is that training turns a pile of random numbers into a structure that can continue a sentence, by making it guess and correcting it, trillions of times.

Billions of dials

A model, before training, is a shape without content. The shape is the architecture, which the next essay covers. The content is a very large set of numbers called parameters. In a model of the size I run at home there are about seventy billion of them. At the start they are set to small random values. The model is, at that point, worthless. Ask it to continue “The Golden Gate Bridge is painted” and it will produce noise.

Think of each parameter as a dial. Training is the process of turning seventy billion dials, by tiny amounts, in the directions that make the model’s guesses better. Nobody turns them by hand, and nobody decides what any single dial should be. The procedure decides, and the procedure has three parts.

An objective. For most of training the objective is to predict the next token. The model is shown a passage from the corpus, one token at a time, and at each position it produces a probability for every token in its vocabulary. The correct answer is simply what came next in the text. If the passage reads “The Golden Gate Bridge is painted international orange,” then at the word “painted” the model should have put high probability on “international.” The gap between the probability it assigned and certainty is its error on that token. Add the errors across the passage and you have a single number, the loss, which says how wrong the model was.

A gradient. Here is the mathematical idea that makes the whole enterprise possible. For every one of the seventy billion dials, it is possible to compute which direction would have reduced the loss, and by how much, for the passage just seen. The method is called backpropagation, and it works by running the error backward through the same arithmetic that produced the prediction. The result is a list, one entry per parameter: nudge this one up a little, that one down a little.

An update. Each dial is turned a small step in its suggested direction. Then the next passage is shown and the whole cycle repeats. The step is kept small on purpose, because a passage about bridge paint should not undo what the model learned from a passage about tax law. Over trillions of tokens, the small steps accumulate into a structure that reflects the regularities of the entire corpus at once.

That is the mechanism. There is no moment at which a fact is written down. The sentence about the bridge leaves behind no record of itself, only a slight change in the odds the model assigns to “international” after “painted,” and to a vast number of other continuations the sentence happened to touch.

The day the chain closed

Every link in this chain existed for decades before they joined. Backpropagation was published in 1986. Graphics cards had been multiplying grids of numbers for games since the 1990s. Large labelled datasets were being assembled by the late 2000s. The moment they came together in public was 2012, at the University of Toronto, when Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton trained a deep network on two consumer graphics cards for five or six days and entered it in the ImageNet competition, which asked systems to name the object in a photograph. The machine that did the training sat in Krizhevsky’s bedroom. Their error rate was fifteen percent. The next best entry was twenty-six. In a field used to gains of a fraction of a point, that gap read as a different kind of result, and the direction of research changed within a year or two.

I keep returning to that result because it is the origin story in miniature: sand, refined into a card sold for playing games, running arithmetic that a method from the 1980s told it how to run, on a million photographs people had labelled, until the guesses were right. Nothing in it was new. The joining was.

The scale of it

The arithmetic is simple and the quantity is not. Meta reported that its largest Llama 3 model was pretrained on roughly fifteen trillion tokens using about sixteen thousand GPUs, and that over fifty-four days the run was interrupted hundreds of times by hardware failures, most of them GPUs and their memory. I quote those figures because they are among the few that a lab has published in full. They describe an operation that is industrial in every sense: supply chains for the chips, power contracts for the buildings, teams on shift to restart a run when a card dies at three in the morning, and engineers deciding how to split one model across thousands of machines so that each sees a piece of the work and they all agree on the answer.

I have come to think of a run like that as one of the most complex things people have built together. That is my assessment, from the outside, and I offer it as such. What is measurable is simpler and still remarkable: a few months of continuous arithmetic across a building full of silicon, after which a file of numbers exists that did not exist before.

Two kinds of training

What I have described is pretraining, and it produces a model that is very good at continuing text and not very good at being asked things. Ask it a question and it may continue with three more questions, because that is what often follows a question on the web. Turning that into an assistant is post-training, and the popular image gets this part half right.

The Shoggoth cartoon shows a smiley mask strapped onto the pretrained monster, as if post-training were paint. In fact post-training changes the parameters too. It is the same mechanism, loss and gradient and update, applied to different data and different objectives. Supervised fine-tuning shows the model demonstrations of good answers and trains it to produce them. Preference optimization shows it pairs of answers that people ranked and trains it to favor the preferred ones, by reinforcement learning in some systems and by more direct methods in others. More recent work trains on tasks where the answer can be checked automatically, such as mathematics or code, and rewards the model for getting there. Which of these a given system uses, in what order and with what data, varies from lab to lab, and most of the specifics are private.

Two things are true at once. Post-training is small next to pretraining, measured in tokens, so most of what the model knows comes from the corpus. And post-training is not a mask, because the dials move. The assistant you talk to is a different set of numbers from the pretrained model, shaped by thousands of human judgments about what a good answer looks like. Those judgments are another place where people enter the story, and the final essay returns to them.

What comes out

At the end there is a file. It holds the architecture’s configuration and the final value of every parameter, and it is large: for a seventy-billion-parameter model, on the order of a hundred and forty gigabytes at the sixteen-bit precision models are usually released in. That file is the model. It can be copied, and the copy is identical. When I run a model on my own machine, I am loading such a file and letting the chip do its multiplication.

From this point on, the dials stay where training left them. Nothing you type at a deployed model turns them. What changes when you ask a question is a different set of numbers, computed fresh for each prompt and discarded after. The distinction between the fixed parameters and the passing computation is the key to the essays on models and inference, and it is worth holding onto now.

The painting

The second panel shows a Toronto lab in the winter of 2012, a reconstruction of the result rather than a record of a moment. Three people and a result. Krizhevsky at the machine, Hinton reading the chart on the wall, Sutskever at the window with the snow behind him. In the foreground, the chain laid out as a still life: a bowl of sand, a quartz crystal, a printed wafer, a graphics card. The chart shows the one thing that matters, a line that had been falling slowly for years and then dropped. The room is modest because the room was modest. The scale of what happened in it is carried by the light. It is the only panel in the series set away from the Bay, and it earns the exception.

The next essay opens the shape itself. The parameters that training sets belong to an architecture, and the architecture that most of today’s language models share is the transformer. It is where “word prediction” stops being a sufficient description and the idea of meaning in context begins.