Short answer: use RAG when the model needs to know facts — especially facts that change or must be cited. Use fine-tuning when the model needs to learn a behaviour — a format, a style, a narrow skill — or when you want a smaller, cheaper or private model to do a task well. Many production systems use both.
The core difference
Retrieval-augmented generation (RAG) leaves the model unchanged. At question time, the system searches your content, retrieves relevant passages and asks the model to answer using them. Knowledge lives in your documents, so updating it is as simple as updating the documents.
Fine-tuning changes the model’s weights by training it on examples of inputs and desired outputs. The model internalises patterns: how to format output, which categories to choose, what tone to use, how to perform a specialised task.
Side-by-side
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Knowledge, facts, documents, policies | Behaviour, format, style, narrow skills |
| Keeps up with changes | Immediately, by updating content | Requires retraining |
| Citations | Natural — answers point to sources | Not inherent |
| Hallucination control | Strong when retrieval is good | Doesn’t stop invented facts |
| Upfront effort | Ingestion and retrieval tuning | Dataset creation and training |
| Running cost | Longer prompts (retrieved context) | Shorter prompts; can use smaller models |
| Data needed | Your documents | Hundreds to thousands of quality examples |
| Permissions | Can filter per user | Everything in the weights is available to all users |
When RAG is the right choice
- An internal assistant answering from policies, manuals and wikis.
- Customer support grounded in your help centre and order data.
- Any case where answers must be traceable to a source.
- Content that changes weekly or monthly.
- Different users should see different information.
When fine-tuning is the right choice
- Classifying or extracting at high volume where a small fine-tuned model can replace an expensive large one.
- Outputs must follow a precise, complex format every time.
- A distinctive style or tone that prompting can’t reliably achieve.
- Domain-specific reasoning patterns, given lots of high-quality examples.
- Privacy or latency requirements that call for a small self-hosted model.
The combined pattern
A common mature architecture: retrieve relevant facts with RAG, and generate with a model fine-tuned to your output format and tone. Fine-tuning can also improve the retrieval side — for example, fine-tuning an embedding model on your domain vocabulary.
A sensible order of operations
- Prompt well and measure on a real evaluation set.
- Add RAG if knowledge is the gap.
- Fine-tune if behaviour, cost or privacy is still the gap — using the evaluation set and production logs you’ve accumulated as training and test data.
Skipping straight to fine-tuning often means training a model to reproduce answers that a good retrieval step would have given you for free.
Cost comparison in practice
RAG’s main costs are building good ingestion and retrieval, plus somewhat longer prompts per query. Fine-tuning’s main costs are building the dataset and evaluation, plus serving and periodically retraining the model. See our guides on RAG chatbot costs and fine-tuning costs for typical ranges.