What is a transformer in AI?

Where the transformer came from

The transformer was introduced in June 2017 in a paper titled "Attention Is All You Need", written by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin, then at Google Brain and Google Research. The paper's opening claim was deliberately radical: "We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely."

Before 2017, the dominant tools for handling sequences — sentences, time series, audio — were recurrent neural networks (RNNs) and their gated variants such as the LSTM. These read data one element at a time, carrying a hidden "memory" forward. That sequential dependency was their fatal weakness: it made training slow and made it hard to connect words that sat far apart in a sentence.

The transformer's authors showed that a model built purely on attention could beat those systems on machine translation while training in a fraction of the time. On the standard English-to-German benchmark the model reached 28.4 BLEU, a new state of the art at the time. That result, and the architecture's scalability, made it the foundation of the modern AI era.

Self-attention explained in plain English

The heart of the transformer is self-attention. Imagine the sentence "The animal didn't cross the street because it was too tired." To understand the word "it", a model must know that "it" refers to "the animal", not "the street". Self-attention is the mechanism that lets the model make that link directly.

For every word, the model produces three vectors: a query (what am I looking for?), a key (what do I contain?) and a value (what do I pass on?). It compares each word's query against every other word's key to produce attention weights — numbers that say how much each word should "pay attention" to each other word. It then blends the values according to those weights. In our example, the word "it" would place a large weight on "animal".

The original transformer runs this process several times in parallel through multi-head attention. Each "head" can learn a different kind of relationship — one might track grammatical subjects, another might track long-range topic. The results are combined, giving the model a rich, multi-angle view of how the sequence hangs together.

Crucially, this comparison happens for all word pairs at the same time. There is no left-to-right bottleneck, which is what makes attention both powerful and fast.

Tokens, embeddings and positional encoding

A transformer does not read raw text. Input is first broken into tokens — words or, more often, word fragments — by a tokenizer. Each token is mapped to an embedding, a list of numbers (a vector) that places the token in a high-dimensional space where similar meanings sit close together.

Self-attention, by design, has no built-in sense of order: to it, a set of tokens is just a set. To restore word order, the transformer adds positional encoding to each embedding — a signal that tells the model where in the sequence each token sits. Without it, "dog bites man" and "man bites dog" would look identical.

These embeddings then flow through a stack of identical layers, each containing an attention sub-layer and a small feed-forward network, knitted together with residual connections and normalisation. Stacking many such layers is what gives large models their depth and capacity.

Encoder, decoder, and the modern split

The 2017 design had two halves. The encoder reads the whole input and builds a rich representation of it. The decoder generates the output one token at a time, attending both to what it has produced so far and to the encoder's representation. This encoder-decoder shape suited translation, the paper's original task.

Modern systems typically use only one half. Decoder-only models — the GPT family, Claude, Llama — are trained to predict the next token and excel at open-ended generation. Encoder-only models such as Google's BERT read text in both directions at once and are strong at understanding tasks like classification and search. Encoder-decoder models such as T5 keep both halves for sequence-to-sequence work.

This flexibility is a large part of why the transformer won: one architecture, sliced three ways, covers most of what modern AI does.

Why it beat RNNs

Two properties explain the transformer's dominance. The first is parallelism. Because attention processes all positions simultaneously, training scales efficiently across GPU and TPU hardware, whereas an RNN must wait for step one before it can compute step two. This let researchers train on far more data, far faster.

The second is long-range context. In an RNN, information about a word early in a passage has to survive being passed through every intermediate step, and it tends to fade. Self-attention connects any two positions directly, in a single operation, so a transformer can relate the first and last words of a long document as easily as neighbours.

The main cost is that standard self-attention scales quadratically with sequence length — doubling the context roughly quadruples the compute. Much subsequent research (sparse attention, FlashAttention, long-context variants) has focused on softening that cost while keeping the benefits.

Beyond text: vision, audio and science

Although born for translation, the transformer turned out to be a general sequence processor. The Vision Transformer (ViT) chops an image into patches, treats each patch like a token, and applies the same attention machinery to achieve state-of-the-art image classification. Speech and audio models use the same idea on sound.

The architecture also drives scientific tools: DeepMind's AlphaFold uses attention-based components to predict protein structures, and diffusion image generators and multimodal models that mix text, images and audio all lean on transformer blocks.

This universality is why the transformer is often described as the substrate of the current AI wave. Almost every foundation model — the large, broadly capable systems that adapt to many tasks — is a transformer at heart.

Frequently asked questions

What does the T in GPT stand for?

The T stands for Transformer. GPT means Generative Pre-trained Transformer: a transformer network, pre-trained on large amounts of text, that generates output. The name reflects the fact that the transformer architecture from the 2017 "Attention Is All You Need" paper is the underlying design.

Who invented the transformer?

The transformer was introduced by eight researchers at Google — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin — in the 2017 paper "Attention Is All You Need".

What is self-attention in simple terms?

Self-attention lets every word in a sentence look at every other word and decide how relevant each one is to it, all at once. This is how a model works out, for example, that the word "it" refers back to "the animal" rather than "the street". Those relevance scores are then used to blend information across the sequence.

Is a transformer the same as a large language model?

No. The transformer is the architecture — the blueprint. A large language model (LLM) is a specific, very large system built using that architecture and trained on text. Every mainstream LLM is a transformer, but the transformer is also used for images, audio and protein folding, so the two terms are not interchangeable.

Why are transformers so fast to train compared to older networks?

Older recurrent networks processed a sequence one element at a time, so each step had to wait for the previous one. Transformers compute attention over the whole sequence in parallel, which maps far better onto modern GPU and TPU hardware and allows training on much larger datasets.

What is the main limitation of transformers?

Standard self-attention scales quadratically with the length of the input: doubling the context length roughly quadruples the compute and memory needed. This makes very long contexts expensive, which is why much recent research focuses on more efficient attention variants.

Related

Explainers: LLM, Foundation model

Glossary: Attention / self-attention, Neural network, Large language model (LLM), Foundation model, Embedding (vector)

Sources

• arXiv — Attention Is All You Need (Vaswani et al., 2017)
• Google Machine Learning Glossary — Transformer
• IBM Think — What is a transformer model?
• Google Research Blog — Transformer: A Novel Neural Network Architecture