Fine-Tuning vs RAG: A Decision Guide With Real Costs

Fine-tuning vs RAG comes down to what you need to change. Fine-tuning changes the model’s weights, so it shapes behavior: format, tone, labels. RAG changes what the model reads at answer time, so it supplies facts that stay current, citable, and deletable. For knowledge, retrieve. For behavior, consider tuning.

Two more options belong in the comparison: long context with prompt caching, and tool calls into systems of record.

What fine-tuning, RAG, long context, and tool calls each change

Each option changes a different part of the system. That is the whole decision.

Fine-tuning changes the weights. You train on labeled examples, and the model’s parameters shift toward producing outputs like them. Google’s supervised tuning docs describe it as adjusting weights to minimize the gap between predictions and labels. They list the right targets: specific output structures, terse versus verbose behavior, and behavior that is hard to describe in a prompt. Nothing in the output shows where a fact came from.

RAG changes the context. A retriever searches your documents, picks a few chunks, and pastes them into the prompt. The weights never move. Update a document, re-index it, and the next answer reflects it. The model can quote the chunk, so you can cite it.

Long context with caching also changes the context, without the retriever. You put the whole knowledge base in the prompt and cache it, so later requests read it at a discount. Anthropic’s contextual retrieval write-up puts the line plainly: if your knowledge base is under 200,000 tokens (about 500 pages), you can just include all of it.

Tool calls change the source. For facts in a database (an order status, a balance, a price), the model calls an API or runs a query. If a number changes hourly, neither weights nor documents should hold it.

My rule of thumb: weights for how to answer, context for what is true, tools for what is true right now.

What does the research say about fine-tuning vs RAG for knowledge?

In published tests, training new facts into a model has mostly lost to retrieval, for the models and methods tested.

Ovadia et al. (published at EMNLP 2024) compared unsupervised fine-tuning with RAG on 7B models. On a current-events test built from August to November 2023 news, Mistral 7B scored 0.481 with no help, 0.588 after its best fine-tuning run, and 0.875 with RAG. The best fine-tuning plus RAG run scored 0.830, below RAG alone. The authors conclude that the models they tested “struggle to learn new factual information” through unsupervised fine-tuning, and that training on many paraphrases of each fact helps.

A newer study adds nuance. Abonizio et al. (August 2025, preprint) injected recent news into Llama-2-7B-chat. Plain continued training gave “modest improvements,” and forgetting rose with training tokens. But one augmented training method (RTW in the paper) reached accuracy “comparable to document-level RAG.” RAG also had a cost: it degraded performance on unrelated control tasks more than every method except instruction pre-training. The authors suggest “the retrieved context may confuse the model on out-of-domain tasks.” Retrieval is not free either.

On long context versus RAG, Li et al. (EMNLP 2024 industry track) found long context beat RAG on average when resources allowed, while RAG cost far less. The two gave identical predictions on 63% of queries. Their router sends each query to RAG first and escalates only when the model says the chunks are not enough. That cut token use to about 61% of full long context for GPT-4o and about 39% for Gemini-1.5-Pro.

One caveat: most of this work uses 7B models from 2023 and 2024. The exact gaps on current frontier models are unmeasured in public work I could find. My read: default to retrieval for facts, and treat any fine-tuned design’s factual accuracy as unknown until your own eval measures it.

Which providers still offer fine-tuning in October 2026?

OpenAI is restricting self-serve fine-tuning in stages. Per its deprecations page, since May 7, 2026, organizations that never ran fine-tuning cannot create jobs. Since July 2, 2026, organizations without fine-tuned inference in the past 60 days cannot either. On January 6, 2027, active customers lose job creation too. Fine-tuned models keep serving until their base model retires, and gpt-4.1 and gpt-4.1-mini retire on January 14, 2027. o4-mini shuts down on October 23, 2026.

Claude has no current fine-tuning path that I can find. On Amazon Bedrock, the fine-tuning table still lists only Claude 3 Haiku, in us-west-2. The Claude 3 Haiku model card shows that region reached end of life on September 10, 2026. I found no current Claude model you can tune through a self-serve service.

Google still sells it. Vertex AI supports supervised tuning on Gemini 2.5 Pro, 2.5 Flash, 2.5 Flash-Lite, and, in public preview, Gemini 3.5 Flash and 3.1 Flash-Lite. On Google’s pricing page, Gemini 3.5 Flash training costs $0.01 per 1,000 training tokens (dataset tokens times epochs), for supervised or reinforcement learning tuning. A tuned Gemini 3 or later endpoint costs 1.5 times the base model to call, and tuned models serve only from the us and eu multi-region endpoints.

If your plan depends on fine-tuning, the provider choice is made for you, and your tuned model retires on someone else’s schedule. Price in a re-tune for every base model retirement.

Fine-tuning vs RAG: a decision table by requirement

If two rows disagree, the stricter one wins.

The permissions row is easy to miss. If a sales rep and an HR partner query one shared cache, both draw on every document in it. On deletion, the UK ICO’s generative AI consultation response (May 2024) says machine unlearning arguments “mainly pointed to its theoretical application,” that output filters “do not actually remove the data from the model,” and that requests need case-by-case review.

Retrieval design (chunking, hybrid search, reranking) is covered in my vector database guide for production RAG. When retrieval needs several lookups, see agentic RAG vs traditional RAG.

What does each option cost? A worked monthly comparison

This is a hypothetical. Prices are Gemini 3.5 Flash on Vertex AI’s global endpoint, from Google’s pricing page on October 15, 2026.

The workload. An internal policy assistant. The knowledge base is 150,000 tokens and changes weekly. Users ask 30,000 questions a month. Each question is 100 tokens, and the system prompt is 1,500 tokens. Each answer bills 400 output tokens in total, including any reasoning tokens, because Google bills text output as “response and reasoning.”

Prices used. Global endpoint: input $1.50, cached input $0.15, and output $9.00 per million tokens. Cache storage $1 per million tokens per hour. Tuned models serve from us or eu endpoints, where base rates are $1.65 and $9.90, so the tuned rates are $2.475 input and $14.85 output. Training costs $10 per million training tokens.

Design A, RAG. Five chunks of 800 tokens per question, so 5,600 input tokens. Per question: $0.0084 input plus $0.0036 output, or $0.012. The index is a $50 monthly placeholder (my assumption). Embeddings cost under $1 a month at Google’s $0.15 per million tokens.

Design B, long context with caching. The system prompt and full knowledge base (151,500 tokens) sit in an explicit context cache that each request references by name. Its default lifetime is 60 minutes, so you extend it. Per question: $0.0227 cached read, $0.0002 for the question, and $0.0036 output, or about $0.0265. Keeping the cache 730 hours a month costs $110.60. A weekly cache rebuild adds about $1.

Design C, fine-tuning. Each weekly retrain uses 3,000 question-answer pairs of 600 tokens each over three epochs: 5.4 million training tokens, or $54 a run. Per question: 1,600 input tokens at $2.475 ($0.00396) plus 400 output at $14.85 ($0.00594), or $0.0099.

How to read it. Fine-tuning has the lowest per-question cost, so at high volume it gets cheaper than RAG. Counting tokens plus infrastructure (RAG’s $50 index included), it beats RAG above about 87,500 questions a month with weekly retrains, or about 1,900 with monthly retrains. Counting tokens and training only (index excluded), the thresholds are about 111,300 and 25,700. The research above favors retrieval for facts, and the tuned model cannot cite its sources. Treat design C’s accuracy as unknown until your eval set measures it.

Sensitivity. At 3,000 questions a month, token and infrastructure costs are $86, $191, and $264. Cache storage dominates design B at low volume. At 300,000 questions they are $3,650, $8,054, and $3,204. Caching matters most: design B without caching costs $6,930 a month in tokens at 30,000 questions, about 7.7 times the cached total. Grow the corpus to 600,000 tokens and design B would cost about $3,260 a month, if the window even allows it. That is past the point where I’d run long context.

The upkeep hours and the $95 loaded hourly rate are my placeholders. Use your own, because they decide the answer more than tokens do. For cutting the token line itself, see how to reduce AI agent token costs.

Failure modes, and how to detect them early

Each option fails quietly in its own way. Build the check before you ship.

Fine-tuning: confident wrong facts. The model answers in perfect format with a fact it half-learned or invented. Detect it with a held-out set of questions whose answers appear only in recent documents. Score facts separately from format, because format scores look great and hide the problem.

Fine-tuning: forgetting and drift. The tuned model loses skills the base model had. Run a small control set of general tasks before and after every tuning job. A drop is your signal to cut epochs or add general examples.

Fine-tuning: base model retirement. Your tuned model stops serving when its base model retires. Track every base model’s retirement date in the same place you track certificate expiry.

RAG: retrieval misses. The right chunk exists but never reaches the prompt, and the model answers from memory. Log retrieved chunk IDs for every answer. Measure recall on a labeled question set: does the correct chunk appear in the top results? Check this first, because generation cannot fix a chunk it never saw.

RAG: distractor chunks and stale indexes. Near-miss chunks can pull answers off course, and Abonizio et al. saw retrieved context lower scores on unrelated tasks. Ingestion jobs also fail silently. Alert on index age, and sample answers where the cited chunk does not support the claim.

Long context: accuracy fading with size. Accuracy on a 50-page corpus tells you little about a 400-page one, because every added page is more near-miss text. Retest your question set whenever the corpus grows meaningfully. My post on context engineering for AI agents covers trimming.

Long context: cache expiry and misses. An explicit cache expires when its lifetime runs out. Then your code has to recreate it or send the full context uncached, which turns a $0.0265 question into a $0.231 one. Alert before expiry and log every uncached fallback. Google’s default implicit caching works differently: it matches repeated prompt prefixes, so a timestamp at the top of the prompt breaks the match.

Agent evals in CI shows how to make these checks block a release.

Bottom Line

Fine-tuning changes how a model answers. RAG and long context change what it knows at answer time.

For facts that change, need citations, or may need deletion, use RAG, or long context with caching if the corpus is small.

Fine-tune only for format or behavior that prompts cannot hold, and check that your provider still offers it.

Next step: write 50 real questions with known answers and run all three designs against them before you pick.

FAQ

Does fine-tuning reduce hallucinations?

It can increase them when the training data holds facts the model did not already know. Gekhman et al. (EMNLP 2024) found models learn such examples much more slowly than familiar ones. Once learned, those examples linearly increased the model’s tendency to hallucinate. Their reading is that fine-tuning mostly teaches a model to use knowledge it already has.

Is RAG plus fine-tuning better than either alone?

It can be, and training for retrieval can improve the combination further. In RAFT (Zhang et al., 2024), Llama-2-7B scored 58.8 on PubMed with RAG and 59.7 with domain fine-tuning, but 71.6 with both. RAFT, which trains on questions paired with useful and distractor documents, reached 73.3. On HotpotQA, GPT-3.5 with RAG still scored higher, 41.5 against 35.28.

Does GDPR erasure apply to a fine-tuned model?

Possibly. The European Data Protection Board said in December 2024 that a model trained on personal data is anonymous only if it is very unlikely that anyone can identify the people in its training data, directly or indirectly, or extract their data through queries. Regulators judge that case by case. My read: if you cannot show that, plan for retraining on request, and keep personal data out of training sets. My post on agent memory and data retention covers deletion in retrieval stores.

How do I make RAG retrieval more accurate before giving up on it?

Fix retrieval before you touch the model. In Anthropic’s tests, adding context to each chunk before embedding cut top-20 retrieval failures by 35%. Adding keyword search (BM25) raised that to 49%, and reranking raised it to 67%. Those numbers come from Anthropic’s own datasets (September 2024), so measure on yours.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect