File MN-004-02 · Field noteLocal time · UTC−3 · Status: Active
Matheus Nugas

Field note 02 · Applied AI

RAG vs CAG: when to stop retrieving and just load it all

Retrieval-augmented generation is the default answer for grounding a model in your data. With big context windows and prompt caching, it is often not the right one.

Filed Reading 5 min

Every time someone wants a model to answer questions about their own documents, the first diagram on the whiteboard is the same: chunk the documents, embed the chunks, store them in a vector database, retrieve the top few at question time. That is RAG, and it works. It is also a lot of moving parts for a knowledge base that would fit in a single prompt. Cache-augmented generation, CAG, is the other option: put the whole knowledge base in the context once, cache it, and ask questions against it.

Two ways to give a model your data

RAG decides what the model sees at question time. A search step picks a handful of chunks it believes are relevant, and the model answers from those chunks only.

QuestionEmbedVector indexTop-k chunksLLM
Fig. 01 · RAG: a search step decides what the model reads

CAG decides what the model sees ahead of time. The whole knowledge base goes into the prompt as a fixed prefix. The model’s processed state for that prefix is cached, so later questions reuse it instead of paying to read everything again. The term comes from a late 2024 paper with a provocative title, Don’t Do RAG, but the idea became practical for everyone once model APIs started offering prompt caching.

Knowledge baseCached prefixQuestionLLM
Fig. 02 · CAG: the model reads everything, once

The difference that matters: in RAG, a wrong retrieval is a wrong answer, and the model cannot tell that it was handed the wrong pages. In CAG there is no retrieval to get wrong. The model has the whole thing.

What RAG costs you

The vector database is the visible part. The real cost is everything you have to tune and keep tuned:

  1. Chunking. Too small and you split the sentence that holds the answer from the sentence that gives it meaning. Too big and the top-k fills up with noise.
  2. Embeddings and ranking. Semantic search is good at “similar” and weaker at “exact”. Product codes, clause numbers and names often need keyword search alongside it, plus a step to merge both result lists.
  3. Questions that span documents. “Which contracts have a renewal clause shorter than the one in our template?” needs most of the corpus, not the five closest chunks.
  4. Re-indexing. Every document change means re-chunking and re-embedding, and a pipeline to do it that you now operate.
  5. Debugging. When an answer is wrong, you first have to find out whether retrieval failed or the model did. That is two systems to inspect for every bug.

None of this is a reason to avoid RAG. It is a reason not to build it by reflex.

What CAG costs you

CAG removes the retrieval pipeline and gives you a different set of constraints:

  1. It has to fit. The knowledge base, the instructions and the conversation all share one context window. That is the hard ceiling.
  2. Long context is not perfect recall. Models are better at using information near the start and end of a long prompt than in the middle. More context also means more irrelevant material competing for attention. Test with questions whose answers sit deep in the document.
  3. Caches expire. Provider caches live for minutes, not days. If traffic is sparse, many requests pay the full price of reading the prefix again.
  4. Caches are exact. The cache matches on the prefix. Change one character at the top of the prompt, and everything after it is new. Anything that varies per request goes after the knowledge base, never before it.
  5. Everyone sees everything. With RAG you can filter retrieval by who is asking. With CAG, everything in the prefix is visible to every question. Different permission levels mean different caches.

In practice, the code is the easy part. With an API that supports prompt caching, CAG is a marker on the big block of text:

const response = await client.messages.create({
  model: 'claude-sonnet-5-5',
  max_tokens: 1024,
  system: [
    { type: 'text', text: INSTRUCTIONS },
    {
      type: 'text',
      text: knowledgeBase, // the whole thing, always in the same order
      cache_control: { type: 'ephemeral' },
    },
  ],
  messages: [{ role: 'user', content: question }],
});

The first request writes the cache. Requests that follow within the cache lifetime read the prefix at a fraction of the normal input price and start answering faster. That is the whole architecture.

How I decide

I ask four questions, in this order:

  1. Does it fit, with room to spare? Leave space for instructions, conversation history and the answer. If the corpus only fits when the context is nearly full, treat it as a no.
  2. How often does it change? A policy manual that changes monthly is ideal for CAG. A ticket database that changes every minute keeps invalidating the cache, so it belongs in RAG.
  3. Is the traffic steady? CAG pays off when many questions hit the same cached prefix while it is alive. A few questions per day means you pay to load everything almost every time.
  4. Does everyone see the same data? One shared corpus is a single cache. Per-user permissions push you toward retrieval with filters, or toward one cache per permission group.

Yes to all four: start with CAG. It is less infrastructure, there are fewer ways to fail, and you can measure answer quality before you build anything else. No to the first question: you need retrieval, at least to narrow the corpus down.

The hybrid is usually the answer

The choice is rarely binary. Most real knowledge bases have a small core that almost every question touches (product rules, definitions, the current price list) and a long tail that only some questions need (old tickets, archived contracts, every past version of a document).

Cache the core. Retrieve the tail. The cached prefix gives the model the context it needs to understand the question, and retrieval only has to find the specific record, which is a much easier search problem than finding everything.

I would rather start with the whole thing in context and add retrieval when the data outgrows it than start with a vector database and spend weeks tuning chunk sizes for a corpus that would have fit in a single prompt.

Filed underApplied AIRAGCAGLLM

Matheus Nugas

Senior fullstack engineer building event-driven systems on AWS. More about me.