Your ticket classifier runs on a 1.6 trillion parameter model. It reads a support message, decides whether it is a billing question or a bug report, and returns one word. You’re paying frontier prices for a job a model a hundredth the size could finish in half the time.

AI model distillation is the standard answer to that mismatch. A large “teacher” model trains a smaller “student” model to reproduce its behavior on a specific slice of work, and the student goes to production at a fraction of the cost and latency. The technique traces back to a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, and small-model releases have quietly leaned on it ever since.

Most introductory guides stop at the teacher and student metaphor, leaving out the two questions that decide whether you should do any of it. The first is licensing. Every major closed-weight vendor sells distillation as a managed product and forbids it in their own terms of service in the same breath, so which license your teacher carries decides whether the pipeline you’re about to build is an optimization or a legal problem. The second question is whether you need a pipeline at all. Somebody has probably already distilled a model close to your task and published the weights, which makes the real decision not how to distill but whether to.

What AI Model Distillation Actually Transfers

Fine-tuning on labeled data teaches a model the right answer. AI model distillation teaches it the teacher’s whole opinion, including the parts where the teacher was unsure. That difference is where the technique earns its keep.

Take a classifier sorting a support ticket. A hard label says “billing.” The teacher’s real output is a probability distribution: 0.82 billing, 0.11 subscription, 0.04 refund, and a long tail of near-zero values spread across every other category. That distribution carries information the label throws away. It says billing and subscription sit close together in a way that billing and password reset do not. Hinton called this the dark knowledge inside a trained network. It explains why a student trained on soft targets learns from fewer examples than one trained on labels alone.

Temperature is the knob that exposes it. Divide the teacher’s logits by a value above 1 before applying softmax and the distribution flattens, lifting those small probabilities to where the loss function can see them. At a temperature of 1 you are nearly back to hard labels. At 5, the student spends most of its learning signal on relationships between categories rather than the winner. The student’s loss then blends two terms: distillation loss against the softened teacher output, and standard cross-entropy against ground truth wherever you have it. DeepInfra’s walkthrough of the distillation training loop goes further into the hyperparameter side.

The payoff predates the LLM era by years. DistilBERT, the 2019 student of BERT, shipped 40 percent smaller while retaining 97 percent of its teacher’s language understanding.

How AI Model Distillation Differs From Fine-Tuning, Quantization, and Pruning

Four techniques get reached for when a model costs too much to run, and they change different things. Distillation trains a new, smaller model to reproduce a bigger one’s behavior. Quantization keeps the same weights and stores them at lower numeric precision. Pruning removes pieces of the network you already have. Fine-tuning changes what a model does without changing its size at all.

The useful part is that they compose. Distill a large teacher into a 14B student, quantize that student down to int8, and serve the result. Each step multiplies against the one before it instead of replacing it, which is why the real question is rarely which technique to pick and usually which order to apply them in.

Four Distillation Tactics, and What Each One Costs You

Most explainers split distillation into response-based, feature-based, and relation-based buckets. That taxonomy is tidy but useless. What actually constrains you is access. How much of a teacher you are allowed to see, and how many teachers you can get that access to.

Response-based distillation is what most teams mean by the word. Prompt a strong model, keep the outputs, supervise-fine-tune a smaller model on the resulting pairs. It needs nothing but an API key, which makes it both the most common approach and the most legally exposed.

Logit matching is the original technique and still the richest signal per example: the student sees the whole distribution instead of one string sampled from it. It also requires running the teacher yourself, on hardware you control, with a tokenizer the student shares. Closed models are structurally incapable of supporting it.

On-policy distillation repairs a failure mode the first two share. A student trained only on teacher transcripts learns from text the teacher would write, so at inference time it wanders into states its training never covered. On-policy training has the student generate first, then has the teacher grade what it produced. That costs more than it sounds like, because the teacher stays online grading for the length of the run instead of doing one batch pass and going away. DeepSeek used exactly this in post-training to fold several domain specialists into one 1.6T parameter release.

Multi-teacher distillation blends several teachers so the student inherits strengths no single teacher has, which is the access problem multiplied: every teacher in the blend has to be one you can actually run. Xiaomi’s MiMo-V2.5 pairs it with large-scale agentic RL. The reasoning case is the one worth watching: DeepSeek-V3 pulled chain-of-thought behavior out of an R1-series model into a standard chat model, which is how it reasons without a visible scratchpad.

Is AI Model Distillation Legal? Why the Answer Pushes You Toward Open Weights

Three constraints stack up, and they all push the same direction.

The first is contractual. Anti-distillation clauses are standard across closed vendors now. A legal analysis of the OpenAI and DeepSeek distillation dispute notes that OpenAI, Anthropic, Mistral, and xAI all carry terms barring the use of their services or outputs to develop a competing model. No technical barrier enforces any of this. Enforcement runs on pattern detection, cease-and-desist letters, and account termination, which puts a pipeline built against a closed teacher on the uncomfortable footing of working exactly as long as nobody looks at it.

The most cited example is the one no team wants to become. When DeepSeek’s R1 landed in early 2025, OpenAI alleged it had been trained on outputs pulled from o1, a dispute Quanta Magazine walked through in detail. Set aside who is right. The lesson for anyone shipping a product is procedural. When your student model’s provenance rests on an argument, that argument is now permanently attached to your inference stack.

The second constraint is technical. Logit matching, the strongest form of distillation, needs the teacher’s full probability distribution at every token. A chat completion endpoint does not return that, and no closed vendor is going to hand over the weights so you can compute it. Black-box distillation on generated text is all a closed teacher can physically support, and it is the weakest variant on the list.

The third is economic. Running a teacher across hundreds of thousands of prompts is metered when it sits behind somebody else’s API, and fixed when it doesn’t. Roughly, a synthetic training set of 200,000 examples at 1,000 input and 500 output tokens each pushes 200M tokens in and 100M out through the teacher. At DeepSeek-V4-Pro’s list rates that estimates out to about $520 for a single generation pass, and you pay it again in full every time you regenerate the set behind a better prompt. Reach for a bigger teacher and the meter reads differently: the same pass against Kimi K3 at $2.85 in and $14.25 out estimates closer to $1,995, because synthetic-data generation is an output-heavy workload and output is where the frontier tier charges. On weights you downloaded, the same pass costs GPU hours, and the fifth iteration costs no more than the first.

Open weights dissolve all three at once. DeepSeek-V4-Pro and MiMo-V2.5 both ship under MIT licenses. Download them, read their logits, generate as much synthetic data as your GPU budget allows, and deploy the student commercially without asking anyone’s permission.

Read the license anyway, because “open weights” has stopped describing one set of terms. Kimi K3 arrived under a bespoke Kimi K3 License instead of the modified MIT terms that shipped with K2. For a distillation pipeline it grants everything that matters: run, fine-tune, create derivative works, sell copies, and nothing in it restricts training another model on K3 output. What it adds sits downstream of the training run. Operate a Model as a Service business whose revenue clears $20 million over any twelve consecutive months and you owe Moonshot a separate agreement, and past 100 million monthly active users you owe them a credit on your interface. A team distilling a ticket classifier will never touch either threshold. A team reselling inference should read that clause before it reads the benchmark table.

The Honest Pros and Cons

What you gain:

• Much lower cost per request, and a smaller memory footprint to serve it from
• Faster time-to-first-token
• Portability onto hardware you own, including hardware with no egress path
• Behavior you can freeze, instead of having it change under you on a vendor’s schedule

What it costs:

• Narrow competence outside the exact task you distilled for
• An eval suite you now maintain forever
• Every mistake the teacher makes, inherited as ground truth
• A retraining cycle every time the task definition shifts

The most useful published figure comes from Nebius, which distilled a 235B-parameter mixture-of-experts teacher into a small student on a grammar-correction task and published the whole pipeline cost: $4.90 for batch generation, $7.10 for fine-tuning, $3.40 for evaluation. Fifteen dollars and change, end to end. The student came out 3.5 times smaller and generated 2.5 times faster than the baseline.

Read their accuracy result carefully, because that is the honest part. The distilled model scored 72.1 percent against a 69.7 percent baseline, with confidence intervals of ±3.2 and ±3.3 points. Those intervals overlap. The distillation bought a large efficiency win and an accuracy number that is, at best, not worse. Distillation compresses capability. It does not create any.

The failure mode to plan around is narrowness. A student distilled on ticket classification classifies tickets. Hand it a ticket in a language your synthetic data never covered, or a category someone added last week, and it degrades in ways the teacher wouldn’t have.

The second failure mode is inheritance. Whatever the teacher gets wrong, the student absorbs as correct, usually with more confidence and no reasoning trace left to audit.

Distill It Yourself, or Call One Somebody Already Distilled

Before building a pipeline, price the alternative. The open-weight catalog is full of models somebody else distilled, at their expense, and released for you to call.

That settles the licensing question too, and in your favor. DeepSeek-V3 carries reasoning behavior distilled out of the same R1 lineage whose provenance OpenAI contested, so the obvious objection is that calling it inherits the argument along with the weights. It doesn’t, at least not the way building your own would. The license came from DeepSeek under MIT terms, and whatever OpenAI’s claim is worth, it runs against DeepSeek rather than against you. That is the structural difference between calling a distilled model and distilling one: the first puts another party between you and a teacher’s terms of service, and the second leaves you standing on them yourself.

Here is what a modest production workload costs across six size tiers. Assume 50M input tokens and 10M output tokens a month, roughly a classifier handling a few hundred thousand requests. These are list prices from the DeepInfra model pages, multiplied out, so treat the totals as estimates rather than a quote.

Top to bottom, that is a 81x spread. You distilled nothing and built no eval suite. Moving the tasks that tolerate it from DeepSeek-V4-Pro down to Nemotron-3-Nano-30B-A3B captures most of what a distillation project would have delivered, on a Tuesday afternoon.

Testing the cheap tier costs one string change, since DeepInfra serves an OpenAI-compatible Chat Completions API:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["DEEPINFRA_API_TOKEN"],
    base_url="https://api.deepinfra.com/v1/openai",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-14B",
    messages=[
        {
            "role": "system",
            "content": "Classify the support ticket. Reply with exactly one word: billing, bug, feature, or other.",
        },
        {"role": "user", "content": "I was charged twice for the October invoice."},
    ],
    temperature=0,
    max_tokens=4,
)

print(response.choices[0].message.content)
copy

Point the same client at a different model string and you have your A/B. This is the practical form of tiered model routing, where a cheap model absorbs the bulk of traffic and bigger ones handle what it cannot. Qwen’s tiers make the math concrete, since Qwen3-14B lands at $0.12 in and $0.24 out per million tokens, close enough to free that the open question becomes whether accuracy holds, not whether the budget does.

Run that experiment first. If a stock open-weight model at a twentieth of the price clears your quality bar, your distillation project was going to spend weeks reproducing a result already sitting behind an environment variable.

When Distilling Your Own Model Is Worth It

Three situations flip the decision the other way.

The volume floor. Distillation is a fixed cost amortized across requests. At a few million tokens a month the arithmetic never closes. At a few billion, halving your per-token cost pays back the pipeline inside the first month and keeps paying.

A task narrow enough to specialize. Structured extraction against a schema you own, routing across a fixed label set, redaction, classification. Jobs with a small output space and stable definitions are where a small student closes nearly the whole gap to its teacher, because there is only so much behavior to copy. Open-ended generation has no such ceiling, which is why distilled students plateau at it while the teacher keeps going.

A latency or placement constraint no hosted model satisfies. If you need sub-50ms responses at the edge, or inference inside a VPC with no egress path, model size stops being a cost preference and becomes a hard requirement. A distilled student that fits on one GPU may be the only shape that works. Small open models are already engineered for this, and Nemotron 3 Nano pairs a Mixture of Experts design with Mamba layers to keep throughput up at that scale.

If none of the three describes your workload, route instead of train.

Getting Started

The fastest way to settle the distill-or-route question is to run both sides against your own eval set. Every model in the table above is live on DeepInfra with pay-as-you-go pricing and an OpenAI-compatible endpoint, which puts the whole experiment inside a day. The pricing page carries current per-token rates, and the documentation covers streaming, structured outputs, and context caching.

If you distill something and want to tell us what broke, we want to hear it. Reach us at feedback@deepinfra.com, join the community on Discord, or find us on X at @DeepInfra.