RAG vs Fine-Tuning: A Technical Decision Guide

Teams often ask whether to use retrieval-augmented generation (RAG) or fine-tuning. They solve different problems, and the right answer comes from an experiment on your data, not from a rule of thumb.

What each one changes

• RAG changes what the model knows at request time, by adding retrieved text to the prompt.
• Fine-tuning changes how the model behaves, by updating its weights on example inputs and outputs.

Decision table

• Facts change often or must be cited: RAG.
• Need a consistent tone, format, or domain-specific output structure: fine-tuning, or few-shot prompting first.
• Need both current facts and a strict style: RAG plus fine-tuning.
• Small data and a weak baseline: improve prompts and retrieval before training anything.

Always start with a baseline

Measure a strong prompt with a few examples before adding complexity. Many tasks that seem to need fine-tuning are solved by clearer instructions and two or three good examples.

def compare(systems, cases, scorer):
    """systems: {"prompt_only": fn, "rag": fn, "rag+ft": fn}"""
    table = {}
    for name, fn in systems.items():
        scores = [scorer(case, fn(case["input"])) for case in cases]
        table[name] = {"mean": sum(scores) / len(scores), "n": len(scores)}
    return table

When fine-tuning pays off

• You have hundreds to thousands of high-quality input and output pairs.
• The task is narrow and repeated, such as classification or structured extraction.
• You want to move to a smaller, cheaper model while keeping quality.
• Latency or prompt length is a constraint, because fine-tuning can remove long instructions.

{"messages": [
  {"role": "system", "content": "Extract invoice fields as JSON."},
  {"role": "user", "content": "INVOICE 2291  Acme Ltd  Total: 1,240.00 EUR  Due 2026-11-30"},
  {"role": "assistant", "content": "{\"invoice_no\":\"2291\",\"vendor\":\"Acme Ltd\",\"total\":1240.00,\"currency\":\"EUR\",\"due\":\"2026-11-30\"}"}
]}

Training formats differ by provider, so check the current documentation before preparing data.

Costs and risks

• RAG needs an index, an update pipeline, and retrieval evaluation, but updating knowledge is just updating documents.
• Fine-tuning needs a training set, retraining when requirements change, and care that private data does not leak into outputs.
• Fine-tuned knowledge goes stale and cannot cite its source.

Our default recommendation

Prompt first, then RAG for knowledge, then fine-tune only for a measured gap in style or cost. Keep the evaluation set from day one so each step has evidence.

A closer look at how each fails

Choosing between the approaches is easier when you know their failure signatures.

• RAG failures: the right passage is not retrieved, the passage is retrieved but ignored, or the model blends it with its own memory. Symptoms are wrong facts that are present in your documents.
• Fine-tuning failures: overfitting to the training examples, forgetting general abilities, or confident errors on facts it never saw. Symptoms are good style with wrong or outdated content.

Evaluating the retrieval side

Before blaming the model, measure retrieval separately. If the gold passage is in the top eight for 95 percent of questions, generation is the next lever. If it is in the top eight for 60 percent, no prompt will fix the system, and you should work on chunking, embeddings, filters, and reranking.

Data requirements for fine-tuning

• Quantity: quality matters more than volume. Hundreds of clean examples can help for narrow tasks, while broad behavior changes need more.
• Quality: examples must reflect the exact behavior you want. Noisy labels teach noisy behavior.
• Diversity: cover the real input distribution, including edge cases.
• Hygiene: remove personal data you do not want echoed, and hold out a test set that is never trained on.

def split_dataset(rows, test_fraction=0.15, seed=7):
    import random
    rnd = random.Random(seed)
    rows = rows[:]
    rnd.shuffle(rows)
    n_test = int(len(rows) * test_fraction)
    return rows[n_test:], rows[:n_test]     # train, held-out test

Combining both

A common production pattern fine-tunes a smaller model to follow your output format and tone, and uses retrieval to supply facts. This can cut cost and latency relative to a large general model with a long prompt, while keeping answers current. It is also more to maintain, so adopt it only after measuring a gap that simpler options do not close.

Maintenance over time

• RAG: keep the index fresh, monitor retrieval recall, and update chunking when document types change.
• Fine-tuning: retrain when requirements or base models change, and re-run the full evaluation each time.
• Both: keep prompts, data, and evaluation sets versioned in source control.

Decision checklist

• Have we measured a strong prompt-only baseline?
• Do answers depend on facts that change or need citations? Prefer retrieval.
• Is the gap in style, format, or cost rather than knowledge? Consider fine-tuning.
• Do we have clean training data and a held-out test set?
• Can we maintain what we build?

How we can help

We run a two-week bake-off on your own data that compares these options on quality, latency, and cost, and hands you the evaluation harness. Contact us to set one up.