Model distillation is a training technique where a large, capable AI model, called the teacher, helps train a smaller model, called the student, so the student can do a similar job with far fewer resources. Instead of learning only from raw examples, the student also learns from the teacher’s answers, including how confident the teacher was.
That extra detail matters. A big model can take a huge amount of computing power to run, which makes it awkward on a phone or a small device with no fast connection. A distilled student is lighter and quicker, so it can answer on the device itself, use less battery, and respond with less waiting. It won’t match the teacher on everything, but for a focused task, it can come close.
Key Takeaways
- Model distillation, also called knowledge distillation, moves behaviour from a large teacher model into a smaller student model.
- The student learns from the teacher’s full outputs, not just right-or-wrong labels, which gives richer guidance than training data alone.
- Distilled models are often faster, cheaper to run, and easier to use on phones and other small devices.
- Distillation works best for a clear, focused task, and it does not remove the need to check the student’s answers.
- A teacher’s mistakes can be passed on too, so testing the student on real examples still matters.
What is the teacher-student idea in AI?
Think of it like a tutor and a learner, though the comparison only goes so far. The teacher is usually a large model already trained on a great deal of data. It is accurate, but also heavy: it may need powerful servers, lots of memory, and a steady connection.
The student starts smaller. During distillation, it is shown the same inputs the teacher saw, and trained to copy the teacher’s behaviour. The training signal is richer than a normal lesson. In ordinary training, an example might be labelled simply as “cat” or “not cat.” In distillation, the teacher might say, in effect, “this is almost certainly a cat, it looks a bit like a dog, and it is definitely not a car.” Those shades of doubt help the student understand how categories relate.
This is a form of machine learning, and it builds on the same idea behind deep learning: models improve by adjusting themselves, step by step, to reduce their errors. Distillation just changes the target. The target is not only the correct label, but also the teacher’s way of answering.
How does model distillation actually work?
In a typical setup, the teacher is frozen, meaning its settings no longer change. The student is the only model being trained. First, a set of inputs is chosen: real examples, questions, or prompts the student will need to handle later. Next, the teacher produces outputs for those inputs, including its final answer and its scores for different options. Sometimes, in language models, it also produces reasoning steps.
Then the student is trained to match those outputs, while also being checked against the correct answers where they are known. This balance matters: copying the teacher alone would copy its errors, while training only on correct labels would throw away the useful detail in the teacher’s answers. Most practical setups mix the two.
Why distil a model instead of just using the big one?
Big models are often brilliant and impractical at the same time. For the wider picture, see the guide to what artificial intelligence is. Running one can be slow and expensive, especially if millions of people use it every day. A smaller model that handles one job well can be a far better fit.
Speed is the most obvious gain. A student that runs on a phone can answer without sending every question to a distant server, which cuts waiting time and helps when the connection is weak. Cost is another: serving a small model needs less hardware, so each answer tends to be cheaper. Privacy can improve too, because data that never leaves the device is less exposed, though that alone does not make an app private, and its other data practices still matter.
What can go wrong with distillation?
Distillation is not a magic shortcut. The most important limit is that a student inherits its teacher’s weaknesses. If the teacher is biased, confidently wrong on some topics, or weak on a particular kind of input, the student can pick up the same habits, sometimes without it being obvious in simple tests.
Another limit is scope. A student trained to copy a teacher on one task may not generalise to a different task as well as the teacher does. That is why teams usually test distilled models on fresh examples, including awkward edge cases, rather than only on the examples used in training.
Where is model distillation used in everyday products?
You probably meet distilled models more often than you realise, because they are the kind that fits where big models do not. Voice assistants on phones, on-device translation, photo sorting, spam filtering, and quick text suggestions all benefit from small, fast models. In each case, a large model may have been used during development, then a smaller distilled version shipped to users. Teams often pair this with fine-tuning to adapt the student further to their own documents or customers.
How is distillation different from fine-tuning and compression?
These terms overlap, so it helps to keep them apart. Fine-tuning takes an existing model and trains it further on a specific task, so it becomes better at that job. Distillation, by contrast, creates a new, usually smaller model by teaching it to copy a teacher.
Compression is the broader umbrella. It covers any method that makes a model smaller or faster, including removing parts of a model, using fewer bits to store its numbers, and distillation itself. So distillation is one kind of compression, while fine-tuning is a different kind of training step. In practice, teams often combine them: distil a model down, then fine-tune the result for the exact job it will do.
FAQs
Is a distilled model always less accurate than its teacher?
Usually, yes, but not always by much, and not always in a way that matters for the job at hand. On the specific task it was distilled for, a good student can come close to the teacher. On broader tasks outside that scope, though, it will often fall behind. The fair comparison is always on the task you actually need.
Can I distil a model on my own computer?
It depends on the size of the models and the data. Small teaching setups can run on a good laptop, and many learning resources use tiny examples to show the idea. Distilling a large language model, however, still needs serious computing power, a suitable teacher, and careful testing. For most learners, trying a small example first is the sensible route.
Does distillation make a model private or safe?
No, not by itself. Running a model on a device can reduce how much data is sent elsewhere, which helps, but the model may still have learned sensitive patterns, and the app around it may still collect data. Privacy and safety need to be checked separately, for the whole product.
Why do companies distil models if they already have a big one?
Because serving the big one to every user can be slow and costly. A distilled model lets them offer a fast, affordable version for common tasks, while keeping the big model for harder cases or for training the next student.
How do I know if a product uses a distilled model?
Often you cannot tell from the outside, and companies do not always say. Clues include fast on-device answers, features that work offline, and a product described as small, efficient, or optimised for mobile. If it matters, check the product’s documentation, which sometimes names the model and how it was built.
Conclusion: a practical next step
Model distillation is best understood as a practical trade: give up some of the breadth of a huge model in return for a smaller one that is faster, cheaper, and easier to run where people actually use it. It works because the teacher’s full answers carry more guidance than simple labels, and because many everyday jobs do not need a giant model at all.
If you want to go further, a good next step is to try a small, well-documented distillation example with a tiny model, and compare the student against the teacher on fresh examples you did not train on. You will learn more from that comparison, including where the student still struggles, than from any claim about size alone. For background, the IBM guide to knowledge distillation and the UK Government’s AI Insights on model distillation are both clear, authoritative places to start.
