A transformer is a type of AI model that reads a whole passage at once and decides which words matter most to each other, so it can work out meaning from context rather than from word order alone. That weighing-up step is called attention, and it is why transformers now sit behind many chatbots, translation tools, and search features.
Key Takeaways
- A transformer uses attention to weigh how strongly each word relates to the words around it.
- Unlike older models that read one word at a time, a transformer can look at a whole passage in parallel.
- Encoder models are best for understanding text, decoder models for generating it, and encoder-decoder models for turning one sequence into another.
- Transformers are powerful, but they can still misunderstand, invent details, and need a lot of computing power.
- You do not need the maths to use them well; knowing what they are good and bad at is enough.
What does attention actually do?
Think about the sentence “The bank refused the loan because it was too risky.” On its own, “bank” could mean a riverbank or a financial institution, and “it” could point to the bank or the loan. A reader sorts that out instantly from the rest of the sentence. Attention is a model’s way of doing something similar.
When a transformer processes text, it first splits the text into tokens, which are chunks of words or parts of words. Each token gets a number-based representation, and the model adds information about where that token sits in the sequence. Then, for every token, it asks: which other tokens should change how I understand this one, and by how much?
In our example, when the model reaches “it,” it should give a lot of weight to “loan” and “risky,” and much less to “refused” or to the riverbank meaning of “bank.” Those weights are not fixed rules written by a person. The model learns them during training, by seeing huge numbers of examples and gradually adjusting until its guesses match the patterns in the data more often.
Why did transformers replace older sequence models?
Before transformers, the usual approach was a recurrent neural network, or RNN. An RNN read text step by step, carrying a kind of short summary forward as it went. That worked for short passages, but the summary got squeezed and muddled over long ones, a bit like trying to retell a long story after hearing it only once, with no notes.
Transformers changed the trade-off. Because they look at the whole input at once, distant words can still influence each other directly. A word at the start of a paragraph can matter just as much as the word right next to it, if the learned weights say so. That also made training easier to split across many processors at the same time, which mattered a great deal as models and datasets grew.
If you have read our guide to how machines learn from data, you will recognise the same pattern: a better way of representing relationships often matters more than a slightly bigger dataset. For a broader picture, start with our complete guide to artificial intelligence, which puts transformers in context alongside other approaches.
What are encoder, decoder, and encoder-decoder models?
Not every transformer is built the same way. The original design had two halves: an encoder that reads and understands the input, and a decoder that produces the output. Modern models often use only one half, or both, depending on the job.
Encoder-only models, such as BERT, read a sentence in both directions. They are good at understanding tasks: sorting a review into positive or negative, picking out names and places, or finding the exact span of text that answers a question. They are not designed to write long, free-flowing text.
Decoder-only models, such as the GPT family, work the other way round. They predict the next token, then the next, building text one piece at a time, and they can only look backwards at what has already been written. That makes them natural for chat, drafting, and brainstorming, but it also means they cannot peek ahead at a full answer before they start writing it.
Encoder-decoder models use both halves. The encoder understands the input, and the decoder turns that understanding into a new sequence. Translation and summarising are classic examples: the input is a sentence or document in one form, and the output is a different sentence or a shorter version of the same content. Our explainer on how computers understand human language covers the language tasks these models are often built for, and our deep learning guide explains the neural-network ideas underneath them.
Where do you meet transformers every day?
You probably meet them more often than you realise. A chatbot that keeps track of your earlier messages is usually a decoder model at work. A translation app that handles a whole sentence, rather than swapping words one by one, is leaning on attention to keep the grammar and meaning together. Search engines, email sorting, and document summarising tools often use encoder-style models to understand what a piece of text is about before they rank or label it.
What are their limits?
Transformers are not magic, and their limits are worth knowing before you rely on them. First, long inputs cost more. Comparing every token with every other token gets expensive as passages grow, so very long documents may be cut, summarised in chunks, or handled with a more efficient variant. That is one reason a tool can lose track of an early instruction in a very long conversation.
Second, attention is not understanding in the human sense. A model can weigh words beautifully and still produce a confident answer that is wrong, out of date, or based on a pattern in its training data rather than a fact. Always check important claims, dates, and numbers against a trusted source, especially for work, health, or money decisions.
Third, training and running large transformers takes a lot of energy and specialised hardware. Smaller, fine-tuned models are often the sensible choice for a single job, such as sorting messages or summarising short reports. Bigger is not automatically better, and a well-chosen smaller model can be faster, cheaper, and easier to check.
Frequently Asked Questions
Is a transformer the same thing as a chatbot?
No. A transformer is a model design. A chatbot is a product built on top of one, usually a decoder model, plus extra training, safety checks, and a user interface. Two chatbots can use similar transformer designs and still behave very differently.
What does “attention” mean in plain English?
It means the model gives different amounts of weight to different words when it interprets each word. Words that help explain the current word get more weight; words that do not help get less. Those weights are learned from examples, not set by hand for every sentence.
Why is the 2017 paper “Attention Is All You Need” so famous?
That paper introduced the transformer design and showed that attention alone, without the step-by-step reading of older models, could handle translation well. It became the foundation for much of the language AI that followed, even though later models have added many refinements.
Do transformers only work with English?
No. The same design works for many languages, and for data beyond text, such as images and audio. Performance can vary by language, though, depending on how much training data was available for it.
Should I learn to build a transformer myself?
Only if that is your goal. For most readers, it is more useful to understand what transformers do, where they fit, and how to check their output. If you do want to go deeper, start with a small, pre-trained model and a guided course, rather than training a large model from scratch.
Conclusion
A transformer is, at heart, a way of reading in context. By weighing every word against the words around it, it can catch relationships that step-by-step models tended to lose, and that single idea now powers a huge share of everyday AI tools. It is not perfect, and it is not a mind, but it is a useful mental model to carry with you.
As a practical next step, pick one tool you already use, such as a chatbot or a translation app, and try the same request twice: once with almost no context, and once with a clear purpose, audience, and example of the result you want. Compare the two answers. You will see, in miniature, what attention is doing, and you will get better results from the tools you already have.
For a deeper technical overview, see IBM’s explainer on transformer models, and for the three main architectures in more detail, see Hugging Face’s guide to transformer architectures.
