RAG vs fine-tuning: the short answer
If your product has to answer from information the model has never seen, such as your documents, your customers’ records or a policy that changed last week, you have a knowledge problem. Retrieval-augmented generation (RAG) is the usual fix: search your content when a request arrives and give the model the passages to answer from.
If the model has what it needs but answers in the wrong format or tone, or handles the same kind of case inconsistently, you have a behavior problem. Fine-tuning is one fix for that, and rarely the first to try.
Most startup products hit the knowledge problem first. That is why we recommend starting with prompting, structured outputs and retrieval, measured by an evaluation set, and reaching for fine-tuning only when that evaluation shows a repeatable gap in format or behavior.
What RAG and fine-tuning each change
RAG changes the input
With RAG, the model stays as it is. Your application searches an index of your content, picks the passages most likely to hold the answer and places them in the prompt with an instruction to answer from them. The knowledge travels with each request, so updating it means updating the index.
Fine-tuning changes the model
Fine-tuning continues training an existing model on curated input and output examples, adjusting its weights so that a format, style or decision pattern becomes its default. It is good at consistency: always this structure, always these labels, always this register. It is an unreliable way to add facts. A tuned model can imitate its examples without knowing where a fact came from, and it knows nothing that changed after they were collected.
Freshness, permissions and citations
Freshness
When a document changes in a RAG system, you re-index it and the next answer reflects the change. When a fact changes in a fine-tuned model, you rebuild the examples, run the tuning job, re-run the evaluation and redeploy, which is too slow for policies, product documentation or anything your customers edit. With retrieval, freshness becomes a requirement you can write down: how quickly an edit, a deletion or a permission change reaches the index.
Permissions
With retrieval, the permission check happens before the model sees anything. Search is filtered by tenant and by the documents this user may open, so a passage they cannot read never reaches the prompt. A fine-tuned model has no such boundary: whatever went into its examples can resurface for any user, so customer data does not belong in a tuning set shared across tenants.
Citations
Retrieval knows which passages it used, so the interface can link each answer to its sources. Users can check a claim, and your team can debug a bad answer by looking at what was retrieved. Citations do not make an answer correct, but they make a wrong one visible.
One caution: retrieved text is untrusted input and can contain instructions. If your assistant can also take actions, read our guide to AI agent guardrails before you connect any tools.
Latency and cost per request
Retrieval adds its cost inside each request. Fine-tuning adds most of its cost around the model.
What retrieval adds
Embedding the query, searching and, where it helps, re-ranking all happen before the model writes a word, and every retrieved passage adds input tokens. More context means a higher cost per request, a slower first response and, past a point, worse answers, because the passage that matters gets buried. The engineering work is retrieving less but better: chunking that follows the structure of your documents, hybrid keyword and semantic search, re-ranking only where the evaluation shows it pays off, and caching for repeated questions within the same permission scope.
What fine-tuning shifts
A tuned model can work with shorter prompts, and a smaller tuned model can sometimes match a larger general one on a narrow task. In exchange, you take on a dataset to curate, a tuning run whenever the base model or your requirements change, and one more model to version and serve. A tuned model is also tied to its base model: changing provider means tuning again, while a retrieval pipeline behind a model-agnostic layer moves with a configuration change.
Measure before you choose
“RAG or fine-tuning?” is a question an evaluation set answers better than any opinion, including ours.
Build the evaluation set first
Collect real questions with known good answers: ordinary cases, questions that span several documents, ambiguous requests and questions the system should decline because your content does not cover them. For structured tasks, include the exact output you expect. Our AI MVP planning guide covers how to gather these examples.
Score retrieval and answers separately
Ask two questions of every case: did search return the right sources, and was the answer correct and faithful to them? A wrong answer then tells you whether to fix search or the prompt. Re-run the set on every prompt, retrieval or model change, tracking format validity, latency and cost per request beside correctness.
A decision checklist
Work through these in order. The first “yes” usually settles it.
- Does the answer depend on your documents, records or data that changes? Use retrieval.
- Must users see only the documents they are allowed to open, or check answers against sources? Use retrieval with a permission filter and citations.
- Is the output the wrong shape, such as invalid JSON, missing fields or labels outside your set? Use structured outputs with schema validation.
- Is the tone or style off? Rewrite the instructions and add worked examples.
- Does the evaluation still show the same format or behavior error across many cases? Fine-tuning becomes a reasonable experiment, measured against your current baseline.
- Is a large model too slow or costly per request for a narrow, high-volume task? Try a smaller model first; a smaller tuned one is worth evaluating if that still falls short and you have enough good examples.
- None of the above? Ship with prompting and structured outputs, and let the evaluation set tell you when to revisit.
Write down which question drove the decision, and check that it still holds whenever your content or the model changes.
When combining them makes sense
RAG and fine-tuning fix different layers, so some products use both. Picture a triage assistant that sorts incoming cases into a detailed internal taxonomy while answering from documentation that changes every week. If the model keeps misclassifying the same kinds of cases despite better prompts and schema validation, a tuned model can carry the taxonomy while retrieval supplies the facts and citations.
Combining them also combines the upkeep: two pipelines, two things to evaluate and a tuned model to rebuild whenever its base model changes. Add the second technique only when the evaluation shows a gap the first cannot close.
Neither one fixes poor source content. If your documents contradict each other or are out of date, no technique produces reliable answers. Cleaning up the source, and deciding who owns it, is often the most useful work in the project.
Where most startup products should start
For a seed to Series B product, the practical default is a sequence rather than a choice:
- Prompting: precise instructions, worked examples and explicit rules for when to decline.
- Structured outputs: a defined schema, validated before anything reaches your database or your users. Invalid output is retried or rejected, never written.
- Retrieval: when answers depend on your content, with permission-aware search and citations.
- An evaluation set: re-run on every change, tracking quality, latency and cost per request.
This sequence works with Claude, OpenAI, Gemini or open-source models and keeps every option open. If fine-tuning is ever justified, the reviewed, redacted interactions you logged in production become the starting point for its examples, kept apart from the evaluation set.
From decision to build
If retrieval is your answer, our RAG development services cover the ingestion pipeline, permission-aware search, citations and the evaluation set. For drafts, summaries or extraction into your data model, see generative AI development; for an assistant grounded in your knowledge, AI chatbot development; and for adding any of this to software you already run, AI integration.
We build these systems with the Fusebox method. Ingestion, retrieval, the permission filter, the answer composer and the citation renderer are separate modules, each with a written contract and a harness. The retrieval harness is an evaluation set with known answers; the permission harness proves a user never sees a document they cannot open. Modules are built in parallel against their contracts, and nothing is wired in until it passes.
Every AI engagement starts with the two-week AI Proof, which ends with a working demo on your data and a go/no-go.
