RAG vs Fine-Tuning: The Difference and When to Use Each

RAG (retrieval-augmented generation) and fine-tuning both adapt a large language model to your data, but they change different things. RAG changes what the model reads: at answer time, it retrieves relevant passages from your documents and adds them to the prompt, and the model itself stays the same. Fine-tuning changes the model: it trains the model further on your examples, which updates its weights.

We build SereneDB, a database that runs the retrieval step of RAG. Every claim below about models and providers links to their own documentation or to the research paper, and every SQL statement ran on SereneDB 26.11.2.

TL;DR

• RAG adds retrieved text to the prompt at answer time. The knowledge lives in a database you can edit, so a changed fact is a changed row.
• Fine-tuning trains the model further on your examples. What it learns lives in the weights, so changing it takes another training run.
• Use RAG for facts that change, for private or per-user data and when answers must cite their sources.
• Use fine-tuning for behavior: a fixed output format, a tone, a narrow task, or a smaller model that does one job well.
• Use both when you need both: fine-tune for behavior, retrieve for facts.

The difference between RAG and fine-tuning

The difference between RAG and fine-tuning is where the new knowledge lives. With RAG, it lives outside the model, in documents that are searched at answer time and pasted into the prompt. With fine-tuning, it lives inside the model, in weights that a training run has changed. Everything else in the table below follows from that.

	RAG	Fine-tuning
What changes	The prompt: retrieved text is added at answer time	The model's weights, through more training
Where the knowledge lives	In a database or search index you query	In the model's weights
How an update ships	Insert, update or delete a row	Train again and deploy the new model
Cost up front	Embedding and indexing your documents	Preparing examples and paying for training
Cost per question	Retrieval, plus the retrieved text as input tokens	The tuned model's own token rates, often with shorter prompts
Latency	Adds a retrieval step and a longer prompt	No retrieval step; shorter prompts can cut latency
Citations	The answer can name the passage it used	The model cannot point to where it learned a fact
Permissions	Filter what each user may retrieve	One set of weights for every user
Data you need	The documents themselves	Labeled examples of good answers
Best at	Facts that change, private data, large document sets	Format, tone, narrow tasks, smaller models

Sources for the rows: the definitions follow the RAG paper and OpenAI's supervised fine-tuning guide; costs, citations and permissions are covered in the sections below.

Star UsContact Us

What RAG does

RAG retrieves passages relevant to a question and gives them to the model together with the question. Lewis et al. introduced it in 2020 (NeurIPS 2020) as a pretrained sequence-to-sequence model (its parametric memory) combined with a dense vector index of Wikipedia read by a neural retriever (its non-parametric memory). AWS describes the same idea for applications: the model consults a knowledge base outside its training data before it answers, without retraining the model.

A RAG application has two halves. The retrieval half finds the passages; it is a search problem, answered by a search index or a database. The generation half is a normal call to the model with a longer prompt.

What fine-tuning does

Fine-tuning continues the training of a pretrained model on a smaller set of examples for your task, which updates the model's weights. OpenAI's supervised fine-tuning guide describes it as training on example prompts with the answers you want.

Full fine-tuning updates every weight. Parameter-efficient fine-tuning (PEFT) trains a small number of extra parameters instead, which cuts compute and storage (Hugging Face PEFT). LoRA is the most common PEFT method (PEFT method overview): it freezes the pretrained weights and trains small low-rank matrices in each layer. On GPT-3 175B, the LoRA paper reports 10,000 times fewer trainable parameters and a third of the GPU memory of full fine-tuning (Hu et al., 2021).

Where prompt engineering fits

Prompt engineering is the third option, and the providers put it first. OpenAI's model optimization guide starts with evals and prompt engineering, and says good instructions and examples may be all a task needs. Google's tuning guide also suggests prompting first and tuning only when prompting is not enough. RAG is prompt engineering with a search step: the instructions stay fixed, and retrieval fills in the facts for each question.

Cost: training runs vs longer prompts

RAG and fine-tuning spend money at different times. Fine-tuning costs most up front and again on every retraining; RAG costs a little up front and a little on every question.

• Fine-tuning. You pay for training, then for running the tuned model. OpenAI prices training per million tokens and gives fine-tuned models their own input and output rates (pricing). Amazon Bedrock bills training by the tokens processed times the number of epochs, plus monthly storage for the custom model (Bedrock custom models). Microsoft's fine-tuning overview notes that fine-tuning adds training and hosting costs. Each change to the knowledge means another run.
• RAG. You pay to embed and index the documents once, to store them, and then for retrieval on every question. The retrieved passages also go into the prompt, so each question carries more input tokens. Keeping the retrieved set small and relevant keeps that cost down: Microsoft's RAG overview notes that sending whole documents wastes tokens.

Fine-tuning can also lower the cost per question. OpenAI lists shorter prompts and a smaller, cheaper model trained for one task among its reasons to fine-tune (model optimization). At high volume on a stable task, that saving can pay for the training.

Freshness: how a new fact reaches the model

With RAG, a new fact reaches the model as soon as the index can return it; with fine-tuning, it reaches the model at the next training run. A fine-tuned model's knowledge is fixed when its training ends, and the RAG paper names updating the world knowledge of such models as an open research problem.

In SereneDB, a changed policy is one statement:

SQL

UPDATE chunks
SET body = 'Refunds are issued within 7 days of the request.'
WHERE id = 1;

The inverted index publishes new and changed rows with a background refresh, every refresh_interval milliseconds (1000 by default), and VACUUM (REFRESH_TABLE) publishes them at once (visibility and the refresh model). The next retrieval returns the new text, as the walkthrough below shows. Removing a fact works the same way: delete the row. Removing a fact from a fine-tuned model means training it again without that data.

Accuracy: hallucinations, citations and permissions

Neither technique stops a model from making things up. RAG gives the model the right passage to work from and lets the reader check it; fine-tuning teaches the model a style and a task but is a weak way to teach it new facts.

• New facts. Ovadia et al. compared the two for knowledge injection and found that unsupervised fine-tuning helps somewhat, while RAG did better consistently, both for knowledge the model saw in pretraining and for entirely new knowledge; the models struggled to learn new facts through fine-tuning (Ovadia et al., 2023). Microsoft's fine-tuning overview makes the same point for current information: fine-tuning doesn't replace retrieval.
• Citations. A RAG answer can include citations to the passages it used, so users can check the source (AWS). Knowledge in the weights carries no such pointer: the RAG paper lists provenance as an open problem for models that keep knowledge in their parameters.
• Retrieval quality. The model can only use the passages it is given, so a RAG system is as accurate as its retrieval. Questions mix meaning with exact terms: "refund for order 4182" needs the embedding for the intent and the exact order number. Hybrid retrieval runs a BM25 keyword ranking and a vector ranking and fuses them, usually with reciprocal rank fusion (RRF), which combines ranks instead of scores that live on different scales (RRF in SereneDB).
• Permissions. RAG can check access at retrieval time: Azure AI Search supports document-level security trimming, and AWS describes restricting retrieval by authorization level. A fine-tuned model has one set of weights for every user, so it cannot answer one user from documents another user may not see. In SQL, the permission check is a filter in the retrieval query.

Star UsContact Us

When fine-tuning is the better choice

Fine-tuning is the better choice when you need to change how the model behaves rather than what it knows. OpenAI's and Microsoft's guides list these cases (model optimization, Microsoft fine-tuning overview):

• A consistent format or tone, such as JSON with a fixed schema or a house writing style that prompts alone do not hold.
• A narrow task done well, such as classification, extraction or tool calls, where examples show the pattern that instructions describe poorly.
• A smaller, cheaper, faster model trained to do one job that otherwise needs a larger model.
• Shorter prompts: instructions and examples move into the weights, so each request sends fewer tokens and returns sooner.
• Using retrieved context well: Microsoft lists training a model to make better use of retrieved context as a fine-tuning goal, which is how the two techniques meet (see using them together).

Is fine-tuning still a thing? Yes, for these jobs. Google Cloud offers supervised fine-tuning for Gemini models, Amazon Bedrock offers supervised and reinforcement fine-tuning and distillation, and open-weight models are fine-tuned with PEFT methods such as LoRA. OpenAI is the exception: it is winding down its self-serve fine-tuning platform (deprecations; details in the FAQ).

When RAG is the better choice

RAG is the better choice when the problem is knowledge: facts the model does not have, facts that change, or facts not every user may see.

• Facts that change. Product docs, prices, policies, tickets. An edit is live after the next index refresh, with no training run.
• Private and per-user data. The documents stay in your database, and each query retrieves only what that user may read.
• Answers that must cite sources. Support, legal and internal search answers can quote and link the passage they used.
• Large document sets. A search index holds far more text than a prompt can, and each question pulls in only the few passages it needs.
• Many tenants. One model serves every customer, and a tenant filter keeps their documents apart.

For example, a support assistant answers from the current help center, and the day the refund policy changes, it answers with the new policy.

Using RAG and fine-tuning together

RAG and fine-tuning combine well because they fix different things: fine-tune the model for behavior, and use RAG for facts. A common split is a model tuned to answer in your format and tone, fed with passages retrieved from your current documents.

RAFT (retrieval-augmented fine-tuning) goes a step further and fine-tunes the model for RAG itself. Given a question and a set of retrieved documents, the model is trained to ignore the distractor documents that do not help, and to quote the relevant passage from the right one in a chain-of-thought answer. The authors report gains on PubMed, HotpotQA and Gorilla (Zhang et al., 2024).

RAG vs fine-tuning vs prompt engineering

The three are steps on one path rather than rivals. Start with a prompt and an eval set. Add RAG when the model lacks facts. Fine-tune when the model has the facts but still gets the format, tone or task wrong, and keep RAG for the facts.

Which one should you use? A five-question checklist

Answer these five questions about your application:

1. How often does the data change? If it changes faster than you would retrain, use RAG. A fine-tuned model is a snapshot of its training data.
2. Do answers need citations? Then use RAG: retrieval knows which passage it returned.
3. Do users have different permissions? Then use RAG, with the permission check in the retrieval query.
4. Is a fixed output format or tone required? Try a prompt first; if it does not hold, fine-tune.
5. What is the latency or cost budget per question? If long prompts are the bottleneck at high volume, fine-tune a smaller model, and keep retrieval small and relevant.

Many applications answer "RAG" to the first three and "fine-tune" to the fourth. That is the combined setup from the previous section.

Star UsContact Us

The retrieval half of RAG in one SQL query

The retrieval half of RAG is a search query: find the passages that match the question by keyword and by meaning, keep only those the user may read, and return the best few with their sources. In SereneDB that is one SQL statement over one index. Start SereneDB in Docker and connect with psql, or any Postgres client:

BASH

docker run --rm -d --name serenedb -e SERENEDB_HOST_AUTH_METHOD=trust \
  -p 127.0.0.1:7890:7890 serenedb/serenedb:26.11.2
psql -h 127.0.0.1 -p 7890 -U postgres

Chunks, metadata and embeddings in one table

Each row is a chunk of a document: the source file for citations, the team allowed to read it, the text, and its embedding. The embeddings here have three dimensions so the example fits on the page; a real embedding model returns hundreds. One inverted index covers the team (an exact-match filter), the text (BM25 through a stemming dictionary) and the embedding (an HNSW graph with cosine distance) (hybrid search, vector search, text analysis):

SQL

CREATE TABLE chunks (
  id   INTEGER PRIMARY KEY,
  doc  VARCHAR,
  team VARCHAR,
  body VARCHAR,
  emb  FLOAT[3]
);

CREATE TEXT SEARCH DICTIONARY en AS
  split_text(case := 'lower')
  | normalize_tokens('en_US.UTF-8', accent := false)
  | stem_words('en_US.UTF-8')
  WITH (frequency, position);

CREATE INDEX chunks_idx ON chunks
  USING inverted (id, team, body en, emb hnsw (metric = 'cosine'));

INSERT INTO chunks VALUES
  (1, 'refund-policy.md',   'billing', 'Refunds are issued within 14 days of the request.',          [0.95, 0.05, 0.00]),
  (2, 'invoices.md',        'billing', 'Invoices are emailed on the first day of each month.',       [0.10, 0.90, 0.00]),
  (3, 'annual-plans.md',    'billing', 'Refund requests for annual plans go to the billing team.',   [0.85, 0.10, 0.05]),
  (4, 'payment-retries.md', 'billing', 'A card payment that fails is retried after three days.',     [0.60, 0.30, 0.10]),
  (5, 'chargebacks.md',     'finance', 'Refund reversals after a chargeback are handled by finance.', [0.90, 0.00, 0.10]),
  (6, 'api-keys.md',        'support', 'Rotate an API key from the security settings page.',         [0.00, 0.10, 0.90]);

The index's background refresh makes the new rows searchable on its own, every refresh_interval milliseconds (1000 by default), so give it a moment before the first query.

One query: BM25 + vector + filter, fused with RRF

A user on the billing team asks "How long does a refund take?". The query below runs two branches over the same index, each filtered to the billing team: a BM25 branch for the keyword "refund" and a vector branch for the question's embedding ([0.9, 0.1, 0.0] here, which your embedding model would produce). RANK() numbers each branch, so rows that tie on score share a rank, and the outer query sums 1 / (60 + rank) per chunk, which is reciprocal rank fusion with the usual k of 60:

SQL

WITH fused AS (
  SELECT id, RANK() OVER (ORDER BY s DESC) AS rank FROM (
    SELECT id, BM25(chunks_idx.tableoid) AS s
    FROM chunks_idx
    WHERE body @@ ts_phrase('refund') AND team @@ 'billing'
    ORDER BY s DESC LIMIT 20
  ) lex
  UNION ALL
  SELECT id, RANK() OVER (ORDER BY dist) AS rank FROM (
    SELECT id, emb <=> [0.9, 0.1, 0.0]::FLOAT[3] AS dist
    FROM chunks_idx
    WHERE team @@ 'billing'
    ORDER BY dist LIMIT 20
  ) vec
)
SELECT c.doc, c.body
FROM fused f JOIN chunks c ON c.id = f.id
GROUP BY c.id, c.doc, c.body
ORDER BY SUM(1.0 / (60 + f.rank)) DESC, c.id
LIMIT 3;

CODE

        doc         |                           body
--------------------+----------------------------------------------------------
 refund-policy.md   | Refunds are issued within 14 days of the request.
 annual-plans.md    | Refund requests for annual plans go to the billing team.
 payment-retries.md | A card payment that fails is retried after three days.

These three rows, with their file names for citations, are what the application pastes into the prompt. chargebacks.md also mentions refunds and sits close to the question's embedding, but it belongs to the finance team, so the filter keeps it out; without the two team conditions, it takes third place. The filter runs inside both branches, so each branch still returns its best billing rows rather than losing some to a filter applied afterwards.

Star UsContact Us

Change a fact, ask again

The refund window changes from 14 to 7 days. With RAG, that is an update to one row:

SQL

UPDATE chunks
SET body = 'Refunds are issued within 7 days of the request.'
WHERE id = 1;

No retraining and no manual step: the index's background refresh publishes the change on its own, every refresh_interval milliseconds (1000 by default). Run the same retrieval query again a moment later:

CODE

        doc         |                           body
--------------------+----------------------------------------------------------
 refund-policy.md   | Refunds are issued within 7 days of the request.
 annual-plans.md    | Refund requests for annual plans go to the billing team.
 payment-retries.md | A card payment that fails is retried after three days.

The next answer quotes the new policy. A model fine-tuned on the old policy keeps the old number in its weights until it is trained again.

The same from LangChain

The langchain-serenedb package runs this hybrid retrieval from a LangChain vector store: BM25 and vector branches over one index, fused with RRF by default. The LangChain integration post shows the vector store in a RAG pipeline, and SereneDB's AI functions can call embedding and generation models from SQL.

Tested with SereneDB 26.11.2 on 2026-11-11.

FAQ

Is RAG better than fine-tuning?

RAG is better for knowledge, and fine-tuning is better for behavior. For facts that change, private data and answers with citations, RAG wins, and in a direct comparison of knowledge injection, RAG beat unsupervised fine-tuning consistently (Ovadia et al., 2023). For a fixed format, a tone or a narrow task, fine-tuning wins. Many applications use both.

Is fine-tuning still worth it?

Fine-tuning is worth it when prompts cannot hold a format or a task, or when a smaller tuned model can replace a larger one at high volume. It is not worth it for adding facts that change; use RAG for those. Start with a prompt and an eval set, as OpenAI and Google both advise, and fine-tune only when the eval shows a gap.

Which LLMs can be fine-tuned, and can ChatGPT be?

As of November 2026:

• OpenAI (the models behind ChatGPT). Fine-tuning at OpenAI is an API feature, and OpenAI is winding it down: since May 7, 2026, organizations that never fine-tuned cannot start, and active existing customers can create new jobs until January 6, 2027 (deprecations).
• Gemini models on Google Cloud support supervised fine-tuning, including Gemini 2.5 Pro, Gemini 2.5 Flash and Gemini 3.5 Flash (Google Cloud tuning).
• Amazon Bedrock offers supervised fine-tuning, reinforcement fine-tuning and distillation for its supported models (Bedrock custom models).
• Open-weight models can be fine-tuned on your own hardware, fully or with PEFT methods such as LoRA (Hugging Face PEFT).

What is the difference between pretraining and fine-tuning?

Pretraining trains a model from scratch on a large general corpus; fine-tuning continues training an already pretrained model on a small set of examples for one task or domain. The LoRA paper describes this as the usual paradigm: large-scale pretraining on general data, then adaptation to particular tasks (Hu et al., 2021).

How do I fine-tune a model?

Collect example prompts with the answers you want, hold some back for evaluation, train, and compare the tuned model with the base model on the held-out set, as Microsoft advises. OpenAI suggests starting with about 50 examples and rethinking the task or prompt if they make no difference (supervised fine-tuning). Use your provider's tuning service, or PEFT methods such as LoRA for an open-weight model.

References

Read on 2026-11-11:

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020.
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021.
• Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, 2023.
• Zhang et al., RAFT: Adapting Language Model to Domain Specific RAG, 2024.
• OpenAI: model optimization, supervised fine-tuning, pricing, deprecations.
• Microsoft: fine-tuning overview, RAG in Azure AI Search.
• AWS: What is RAG?, Bedrock custom models.
• Google Cloud: tune Gemini models.
• Hugging Face: PEFT, PEFT method overview.
• SereneDB 26.11.2: hybrid search, vector search, reciprocal rank fusion, visibility and the refresh model.

About SereneDB

SereneDB is the first real-time search analytics database: one Postgres-compatible engine that answers hybrid questions, finding the relevant records in unstructured data and computing over them in the same query, built for applications and AI agents that ask both at once. For RAG, that means keyword, vector and filter retrieval in one SQL query over data that updates in place. See the RAG database and hybrid search pages, or the source on GitHub.