Encoders: Translating Experience into Representation

Start with a photograph of a red hibiscus and the phrase a red flower. To a person they obviously belong together. To a computer they have nothing in common. The photograph is a grid of a few million numbers, three per pixel, giving the brightness of red, green and blue. The phrase is three token numbers from a vocabulary. No arithmetic on the first produces the second. If a system is going to treat them as related, something has to carry each of them into a common form where relation can be measured.

That something is an encoder, and the common form is a vector. The previous essay described how a transformer computes a vector for every token in context. This essay steps back to ask what a vector is for, how an image can become one, and what it means when two of them are close.

A position, not a definition

A vector here is a list of numbers, typically a few hundred to a few thousand long. It helps to think of the list as coordinates. A list of three numbers places a point in ordinary space; a list of a thousand places a point in a space with a thousand directions, which nobody can picture and which the arithmetic handles without trouble. Every input the encoder processes becomes a point. The whole space, with all the points training has arranged in it, is the representation.

Two things follow that are easy to get wrong.

First, the numbers mean nothing on their own. The seventh coordinate of the vector for the hibiscus is not “redness” or “petal count.” Meaning lives in the arrangement: which points are near which, along which directions things vary. That arrangement is learned, and it is learned for a task. An encoder trained to match photographs to captions arranges its space one way. An encoder trained to detect manufacturing defects arranges its space another way, and the same photograph lands somewhere entirely different. Take a vector out of one model and hand it to another and it is noise.

Second, there are two different objects that both get called embeddings, and the transformer essay touched both. One is the lookup table: a fixed vector per token, the same every time, learned during training and consulted at the first step. The other is the contextual state: the vector at a position after the layers have mixed in the surrounding text, different for every sentence. The phrase I have heard used for all of this, a library of vectors of meaning with transformers that interpret them in context, is a fair starting picture if the library is understood as the table and the interpretation as the computed state. The refinement is that the library is small and static, the interpretation is where the work happens, and neither contains meaning the way a dictionary does. They contain positions that training made useful.

Encoding an image

Text arrives as tokens. An image has to be made into something token-like first. One approach, the one that won in 2012, slides small filters across the picture and builds up from edges to textures to parts. A newer approach cuts the image into a grid of patches, perhaps sixteen pixels square, treats each patch as a token, and runs the same transformer machinery the previous essay described. Either way, the output is a vector, or a set of them, that stands for the image.

An image encoder trained alone can learn to recognize things. It cannot, by itself, know that its vector for the hibiscus should sit near a text model’s vector for a red flower, because nothing in its training connected the two spaces. Connecting them is a separate act.

Putting two kinds of thing in one space

In 2021, researchers published a system called CLIP that did the connecting directly. It trained two encoders at once, one for images and one for text, on roughly four hundred million image and caption pairs gathered from the web. The objective was contrastive: for each batch, make the vector of every image close to the vector of its own caption and far from the vectors of every other caption in the batch. Nothing told the system what a flower was. It only ever learned that this picture goes with these words and not those, hundreds of millions of times, and the two spaces were pulled into alignment as a result.

After that training, the hibiscus photograph and a red flower land near each other. A blue truck lands far away. A red rose lands close, probably closer than the designers of any rule would have placed it. And the system can do something it was never explicitly taught: given a new photograph and a list of candidate phrases, pick the phrase whose vector is nearest. That is classification without a classifier, and it is one reason systems that turn text into images can understand a prompt at all: a text encoder, in several widely used generators one from this family of models, is what tells the image generator what a red flower at dusk should land near.

Measuring “near”

Nearness needs a number. Two are common. The dot product multiplies the two vectors coordinate by coordinate and adds the results; it grows when the vectors point the same way and when they are long. Cosine similarity divides that out and keeps only the angle, so it ranges from minus one to one regardless of length. For two short illustrative vectors, [0.8, 0.2, 0.6] and [0.7, 0.3, 0.6], the dot product is 0.98 and the cosine similarity is about 0.99. The numbers are close here because the vectors happen to have similar lengths; in general they are different quantities, and a published figure should say which one it is.

What similarity is not

A shared space is a genuine achievement and it is worth being precise about its limits.

Near means near according to the captions the system was trained on. If photographs of red flowers on the web are usually captioned with the word flower and rarely with the word hibiscus, the space will know the first relation better than the second. If the captions carry the habits of the people who wrote them, so does the space. A photograph of a red lamp shaped like a flower may land close to a red flower because that is what the arrangement rewards. There is no coordinate for truth.

Similarity is also not understanding, and I want to hold that line as carefully as I held it for attention. The system can tell you that the photograph and the phrase are close. It cannot tell you what a flower is, in the sense a gardener could. What it has is a geometry in which the two are neighbors, and that geometry is useful for exactly the tasks it was built for.

Finally, not every language model has a separate text encoder, and the systems that take both text and images differ in how they join them. In some published designs, an image encoder’s output is projected into the language model’s own token space and treated as a run of tokens the model can attend to; in others, image features enter the model’s layers through additional attention. For the models whose internals are not public, I do not know, and will not guess.

Translation and loss

Here is the cultural reading, and I offer it as reading. An encoder is a translator, and like every translator it loses. The hibiscus in the photograph was a living plant with a smell, a weight, a morning. The photograph lost most of that. The vector loses most of what the photograph kept, and keeps what the task required: enough to find the phrase, enough to find the rose. What arrives inside the machine is a position, arranged by a history of other people’s captions. That is a thin thing compared to a flower, and it is also the only way anything outside text has ever entered one of these systems. Every image a model has ever “seen” came in this door.

The painting

The fourth panel shows the door. One photograph of a red flower, one card with the phrase, held up by two people on either side of a single diagram: an image encoder on the left receiving the photograph, a text encoder on the right receiving the phrase, two points landing close together inside a drawn circle. The arrows run into the encoders and out to the points, and nowhere else. The diagram will be an exact vector graphic composited into the painting, because an arrow painted the wrong way, as in an earlier sketch, teaches the wrong thing. Beyond the window, the Bay, with Angel Island and the Marin hills as they actually sit from the city’s north side.

The next essay follows the flower inside. Once a model has trained, where is the hibiscus? Where is Huckleberry Finn? The honest answer is that there is no where, and explaining why is the subject of Models.