When an AI system yields unreliable answers, the root cause could be an unclear system prompt, missing context, poor retrieval quality, or simply using the wrong base model. Teams end up spending weeks experimenting with prompt changes, retrieval-augmented generation (RAG), or fine-tuning to improve response quality.

But before deciding which technique to adopt, it is important to understand what problem each approach solves. Prompting changes how the model is instructed, RAG provides external information, fine-tuning modifies repeatable model behavior, and distillation optimizes an AI model for lower cost and latency. 

Treating them as interchangeable techniques increases engineering overhead without improving application performance.

This guide explains when to use prompting, RAG, fine-tuning, or distillation in production. It also shows how to choose the right base model, add RAG, fine-tune behavior, and distill only after the workflow is proven.

Which Approach Should You Use Right Now?

The right choice depends on the application’s primary failure mode. Before adding more complexity, determine whether the problem is unclear instructions, missing knowledge, inconsistent behavior, or production constraints such as latency and cost. 

The table below maps these common failure modes to the most appropriate solution.

How to Improve Production AI Systems

For many production AI systems, a practical approach is to start with the cheapest and fastest option, then add complexity only when evaluation shows that a different solution is needed.

Each stage below addresses a different production problem and lets teams add complexity only when the previous approach is no longer sufficient.

Stage 1: Control the model through prompting

Prompting is the best place to start because it’s the quickest and most easily reversible way to improve model performance. 

You can update the prompt to test the model and roll it back in minutes without changing the model or deploying additional infrastructure. 

It also sets the baseline for measuring future improvements. If better instructions or output constraints solve the problem, there is little reason to introduce retrieval or training.

Stage 2: Supply missing knowledge through RAG

If prompting cannot resolve the problem and the missing piece is knowledge, RAG is usually the next step. If the correct answer exists in private documents, recent data, or user-specific records, retrieving that information at inference time is simpler and easier to maintain. This approach is more efficient than retraining the model every time the knowledge changes.

Stage 3: Change repeatable behavior through fine-tuning

When prompting and retrieval don’t solve the problem, the next step is usually to fine-tune the model. At this point, the team has verified that the model has sufficient knowledge but still makes mistakes, such as formatting inconsistently or choosing the wrong tools. 

Since fine-tuning creates a new version of the model that needs its own testing, deployment, and monitoring, it should be done only when these extra tasks are worth the benefit.

Stage 4: Compress the proven system through distillation

Distillation transfers the behavior of a teacher (a single large model, fine tuned model, ensemble, or reviewed prompt plus RAG pipeline) into a smaller student model. The goal is to keep enough of known-good behavior while lowering latency and cost and making it easier to run on less powerful hardware.

Before compressing a workflow into a smaller model, teams need to be sure that the prompt, retrieval strategy, and model behavior already meet the needs for production. Otherwise, distillation will simply make an immature system cheaper and faster but not better.

When Is Prompting Enough?

Prompting is enough when the model already has the required capability and knowledge but needs clearer instructions, better examples, stronger constraints, or a reliable output format. Before adding RAG or training data, also test whether a different base model solves the failure more cleanly.

Start by choosing the right model

A weak model cannot be prompted into capabilities it does not have. So, before assuming prompting has failed, confirm that the base model is capable of the task. If the application depends on advanced reasoning, tool use, long-context understanding, or multilingual performance, testing a more suitable model may solve the problem.

Improve the prompt before changing the system

A good prompt reduces ambiguity and gives the model a consistent way to approach each request. In production, keep the stable instructions separate from the information that changes between requests. For example, the system prompt can define the model’s role and output format, while the user message supplies the specific question or document to process.

When those stable instructions remain the same across many requests, they can also become an inference cost. DeepInfra prompt caching reuses the previously computed KV (key-value) cache for identical prompt prefixes, reducing latency and cost. 

However, caching should not become an excuse to keep instructions that no longer add value. Remove redundant examples and unnecessary context before optimizing how they are served.

The example below demonstrates how prompt caching works when the same system prompt is reused across multiple requests.

from openai import OpenAI


client = OpenAI(
    api_key="$DEEPINFRA_TOKEN",
    base_url="https://api.deepinfra.com/v1/openai",
)


# Long system prompt that stays the same across requests
SYSTEM_PROMPT = """You are a helpful AI assistant with deep expertise in Python.
[... thousands of tokens of instructions or context ...]
"""


# First request — full processing
response1 = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "How do I use list comprehensions?"},
    ],
)


# Second request — cached prefix for system_prompt reused, faster and cheaper
response2 = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "What are Python generators?"},
    ],
copy

Know when the prompt has reached its limit

Prompting has likely reached its limit when you observe one or more of the following:

• Instructions become increasingly long or contradictory. Adding more constraints creates confusion instead of improving results.
• Additional examples no longer improve performance. The model continues making the same mistakes despite better prompting.
• Small prompt changes cause large output regressions. Minor wording changes produce inconsistent or unpredictable behavior.
• The same task repeatedly fails despite clear instructions. Persistent errors in classification or structured output usually indicate a behavioral limitation rather than a prompt design problem.
• The model lacks the required knowledge. If the information needed to answer the request is not available to the model, prompting alone cannot produce the correct response.

When Should You Add RAG?

Add RAG when the model already has the reasoning ability you need but lacks access to the right information at inference time. This is usually the case when knowledge is private, frequently updated, too large to include in every prompt, permission-controlled, or required to be grounded in verifiable sources.

RAG is well suited for applications that rely on internal documentation, product catalogs, customer records, enterprise knowledge bases, legal and compliance material, technical documentation, or research workflows that require citations. It is also the better default for user-specific or permission-sensitive information because access controls can be applied before the retrieved context is sent to the model.

If opting for the RAG approach, you’ll need a retrieval layer that can consistently surface relevant context. DeepInfra provides an OpenAI-compatible embeddings API and hosted reranker models to help build production-ready RAG applications.

Here is minimal code to generate embedding vectors using the OpenAI-compatible embeddings API. 

from openai import OpenAI


openai = OpenAI(
    api_key="$DEEPINFRA_TOKEN",
    base_url="https://api.deepinfra.com/v1/openai",
)


input_text = "The food was delicious and the waiter..."
# Or a list: ["hello", "world"]


embeddings = openai.embeddings.create(
    model="Qwen/Qwen3-Embedding-8B",
    input=input_text,
    encoding_format="float"
)


print(embeddings.data[0].embedding)
print(embeddings.usage.prompt_tokens)
copy

Code example to rerank a list of documents by relevance to a query.

import requests


DEEPINFRA_TOKEN = "$DEEPINFRA_TOKEN"
MODEL = "cross-encoder/ms-marco-MiniLM-L-12-v2"


response = requests.post(
    f"https://api.deepinfra.com/v1/inference/{MODEL}",
    headers={
        "Authorization": f"Bearer {DEEPINFRA_TOKEN}",
        "Content-Type": "application/json",
    },
    json={
        "query": "What is the capital of France?",
        "documents": [
            "Paris is the capital and most populous city of France.",
            "Berlin is the capital of Germany.",
            "The Eiffel Tower is located in Paris.",
            "France is a country in Western Europe.",
        ],
    },
)


result = response.json()
for item in result["scores"]:
    print(item)
copy

What RAG does not fix

RAG will not fix ambiguous instructions, inconsistent tone, weak tool selection, invalid JSON, or a base model that cannot perform the underlying task. It also cannot rescue retrieval results that are irrelevant, incomplete, stale, or blocked by incorrect permissions.

This is why RAG systems need two evaluation tracks. Retrieval evaluation assesses whether the right evidence was found, while generation evaluation measures whether the model used that evidence correctly. Combining both into a single end-to-end score makes failures harder to diagnose.

When Should You Fine-Tune the Model?

Fine-tune when the task is stable, the behavioral gap is measurable, representative examples are available, and prompting or retrieval cannot close the gap. It works best when the model already knows how to perform the task but cannot do so consistently across similar requests. 

The approach is suited for tasks that involve domain-specific extraction, reliable structured output, tool selection, function argument generation, and repeated transformations. It can also reduce the prompt size because the desired behavior is learned during training instead of being repeated through long instructions in every request.

Do not fine-tune frequently changing facts into application weights

Fine-tuning is effective for learning task patterns and domain language, but it is not ideal for facts that change often or depend on the source. Once data is encoded in the model’s weights, selectively updating individual facts becomes challenging as facts are distributed across weights, and those same weights contribute to many other learned behaviors. As a result, updating one piece of knowledge often requires a completely new training cycle.

This does not imply that knowledge and fine-tuning are unrelated. But a domain-specific fine-tune can significantly refine a model’s grasp of terminology or apply a stable reasoning pattern. The point is operational: information that must remain current, permissioned, independently editable, or directly cited is generally safer to retrieve at inference time.

Use LoRA as the practical starting point for many application fine-tunes

For most production applications, LoRA is a better starting point than full fine-tuning because it adapts a pretrained model by training a small set of additional parameters. The LoRA method makes experimentation faster, reduces GPU memory requirements and training costs, and allows multiple task-specific adapters to be maintained for the same base model.

Full fine-tuning provides deeper control over the model but requires significantly more compute, storage, and operational overhead. It is generally reserved for cases where adapter-based methods cannot achieve the required behavior or when building a highly specialized model.

DeepInfra supports deploying LoRA adapters over supported base models through an OpenAI-compatible API. Teams that need to train or fine-tune their own checkpoints can use dedicated GPU instances and then deploy complete custom weights via private model infrastructure.

Example to deploy LoRA fine-tuned language models on DeepInfra.

Using the public adapter askardeepinfra/llama-3.1-8B-rank-32-example-lora (base: meta-llama/Meta-Llama-3.1-8B-Instruct):

1. Go to Dashboard → New Deployment → LoRA Model
2. Fill in:
   • LoRA model name: asdf/lora-example
   • Hugging Face Model Name: askardeepinfra/llama-3.1-8B-rank-32-example-lora
3. Click Upload

The deployment appears in Dashboard → Deployments. Initial state is Initializing → Deploying → Running.

Once running, your model page is at https://deepinfra.com/asdf/lora-example.

Inference

curl "https://api.deepinfra.com/v1/openai/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPINFRA_API_KEY" \
  -d '{
      "model": "asdf/lora-example",
      "messages": [
        {
          "role": "user",
          "content": "Hello!"
        }
      ]
    }'
copy

When Should You Use RAG and Fine-Tuning Together?

Use RAG and fine-tuning together when the application needs both changing external knowledge and specialized, repeatable behavior. Retrieval supplies evidence, fine-tuning teaches a stable way to interpret or act on that evidence, and prompting controls the current request, policy, and permissions.

For example, in a customer support assistant, RAG collects account info, product details, and refund rules. Finetuning can improve categorization, escalation, response structure, and use of company terms. The prompt defines the channel, tone, prohibited actions, and when a person review is needed. 

When the methods are combined, evaluate them separately. Retrieval metrics should measure whether the right evidence reached the model. Generation metrics should measure whether the model followed the evidence and the policy. Fine-tuning should not be credited for a retrieval improvement, and RAG should not be blamed for a model that ignores a correct passage.

When Does Model Distillation Become the Right Next Step?

Model distillation makes sense after the application works reliably and traffic is high enough for model cost, latency, memory, or throughput to become a business constraint. It is not an alternative to prompting, RAG, or fine-tuning, but it is a way to compress a validated model or workflow into a smaller student.

Distill only after the task and acceptance criteria are stable

Before distillation, you should know which task is stable, what a correct output looks like, which edge cases matter, what quality loss is tolerable, and which latency or cost target the student must meet. Distilling an unstable system does not remove its design issues but makes existing errors cheaper and faster to produce.

Narrow tasks are usually easier to distill than open-ended assistants. Tasks like classification, routing, structured extraction, content moderation, query rewriting, and repetitive agent substeps have clear outputs that can be evaluated at scale. In contrast, a broad system that relies on changing tools, policies, and retrieval behaviors is less stable and more challenging to distill effectively.

Validate the quality-cost trade-off, not just raw model size

A smaller student can reduce inference cost and improve speed, but it may also lose capability or robustness outside the training distribution. Teacher errors can enter the training data, and a benchmark that resembles the distillation set too closely can overstate generalization.

The business metric should be cost per acceptable task. Compare the student and teacher on unseen production examples, difficult edge cases, abstention behavior, and downstream workflow success. Re-run those tests whenever the upstream prompt, retrieval layer, or teacher model changes materially.

DeepInfra can host pre-distilled models and custom weights. You can perform the training yourself and use DeepInfra for the resulting inference workload.

• Note: While we host distilled models, we do not currently provide services to distill models.

How DeepInfra Supports the Production Adaptation Stack

DeepInfra brings the production adaptation workflow together on a single OpenAI-compatible platform. Teams can iterate on prompts across more than 100 open-weight models, build RAG pipelines with hosted embedding and reranking models, deploy LoRA adapters or custom fine-tuned checkpoints, and serve distilled models through private model deployments without managing inference infrastructure. This lets developers move from prompting to RAG, fine-tuning, and distillation without changing platforms as their applications evolve.

Conclusion

Choosing between prompting, RAG, fine-tuning, and distillation depends on your constraints. Prompting controls the current request, RAG provides external information, fine-tuning enforces repeatable behavior, and distillation optimizes proven workflows. The ideal architecture minimizes complexity while hitting quality, latency, and cost targets. Start with the prompt, retrieve what changes, train what repeats, and distill only what has already proven its value.

Explore DeepInfra’s open-weight models to compare pricing, context windows, and performance for prompting, RAG, fine-tuning, and distilled inference workloads.