RAG, fine-tuning or long context: how to make AI answer from your documents

Show the model the right text at the moment of the question. That is the whole idea, and the three approaches differ only in how the text gets there. With long context you paste in everything relevant. With retrieval (RAG) a search step picks the passages for each question. With fine-tuning you try to bake the material into the model beforehand. The first two work by showing. The third works by teaching, and teaching is the least reliable way to get specific facts back out.

The model is not a database. It does not hold your documents, and it cannot be asked to 'look something up' unless something puts the text in front of it. Everything below follows from that. The problems that matter in practice (is the answer current, is the user allowed to see it, can we check it) are about what is put in front of the model and who decided, which is engineering you control, not a property of the model.

If your question is really about structured data, such as how many invoices a supplier sent last quarter, none of these is the answer. That is a database query, which a model can help write and your system should run. Retrieval over documents is for questions whose answers live in prose: policies, contracts, manuals, emails, specifications.

Long context means giving the model the documents themselves, inside the prompt, along with the question. Anthropic's documentation describes the context window as the model's working memory: everything in the request counts toward it, including the system prompt, documents, tool results and the answer itself, and the size varies by model.

For a modest, slowly changing set of documents it is the simplest option, and arguably the first one to try. Anthropic's engineering write-up on contextual retrieval makes that point directly: if your knowledge base is small enough to fit in the prompt, you can include all of it and skip RAG. That post was written for a 200,000-token window (it gives about 500 pages as a rough guide), and current windows on many models are larger, but the logic holds and the arithmetic changes. It adds that prompt caching, which lets repeated prompt content be reused between calls, makes this approach faster and cheaper.

• Strengths: nothing to build, nothing to keep in sync, and the model sees whole documents with their context intact. It is also the quickest way to learn whether the task is feasible at all.
• Limits: it does not scale to a large or growing corpus. Each question pays to send the material again (caching reduces that, it does not remove it). And bigger is not automatically better. Anthropic's documentation says accuracy and recall can degrade as token count grows, which it calls context rot, so curating what goes in matters as much as how much fits.
• Structure helps. Anthropic's prompting guidance for long inputs is to put the long documents first and the question last, wrap each document with its source in tags, and ask the model to quote the relevant passages before answering.
• Permissions are crude. Everything in the prompt is visible to the person asking. If different users may see different documents, you must build a different prompt for each, which is a form of retrieval anyway.

RAG means retrieval-augmented generation: look up the relevant passages first, then ask the model to answer using only those. Anthropic's description is that it retrieves relevant information from a knowledge base and appends it to the user's prompt. Think of an assistant who is handed the three most relevant pages of the manual before they answer, instead of being expected to remember the manual.

In outline, there are two phases. Preparing: split each document into chunks (passages of a few hundred words or fewer), turn each chunk into an embedding (a list of numbers that captures its meaning, so similar passages end up close together) and store them in an index. Answering: turn the question into an embedding, find the closest chunks, optionally re-rank them, and give the best few to the model with the question. Anthropic's documentation notes it does not offer its own embedding model and names Voyage AI as one provider, while telling readers to assess several vendors. Our estimating assistant lists Voyage AI in its stack.

Meaning-based search misses exact strings. Anthropic's write-up gives the example of an error code: an embedding search may find content about error codes in general and miss the one called TS-999, whereas a keyword method such as BM25 finds the exact match. Combining the two is common, and so are part numbers, invoice numbers and clause references in operations documents, so a hybrid is usually worth having. Our estimating assistant scores a new brief on a hybrid of weighted parameters, semantic similarity and equipment-category overlap, and falls back to keyword overlap if no embedding key is set.

RAG is the fit when the corpus is too large for a prompt, changes often, differs by user, or must produce answers that point back to a source. It also has the most moving parts: a pipeline to load, split, embed and update; an index to host; a retrieval step to tune; and a prompt to assemble. Each part can be wrong silently, which is why testing comes before the build, not after.

Fine-tuning means further training a model on examples so that it behaves differently: it follows a house format, adopts a tone, classifies into your categories or handles a narrow task more consistently. It changes the model's habits. It is a poor way to make the model reliably know your documents, for four reasons that follow from how it works rather than from any vendor's limits.

• Facts go stale. A trained model knows your policy as it was on the day of training. Changing one clause means preparing data and training again, whereas a RAG index is updated by changing one document.
• It cannot be permissioned. Whatever the model learned, it can say to any user who asks. You cannot train in a document and then show it to some people only. Retrieval can filter by user before the model sees anything. A tuned model cannot.
• It cannot cite. A fine-tuned answer comes from the weights, with no passage to point to, so a reviewer cannot check it against a source.
• It is tied to a model. When the base model is retired, the tuned version may be too, and the work is repeated. Availability varies by provider and model and it changes, so confirm it in the provider's current documentation or with your account contact before planning around it.

Where it does fit: a stable, narrow, high-volume task where prompts and examples have stopped being enough, such as a fixed output style or classification into a closed set, and where you hold plenty of good labelled examples. Even then, try better prompts and a handful of examples in the prompt first, and measure. Anthropic's documentation describes using prompt caching to include a large set of high-quality example answers in the prompt, which gets some of the benefit with none of the training. Treat that as a pointer rather than a promise about your task.

They combine. A common arrangement is retrieval to supply the facts and, rarely, a lightly tuned model to hold the format. Another is long context for a handful of core documents that every answer needs, with retrieval for the long tail. Start with the cheapest approach that passes your own test and add complexity only on evidence.

A stale index

The answer is wrong because the index still holds last quarter's policy. Nothing errors. The assistant simply quotes the old text with full confidence. The fixes are operational: re-index on change (not on a monthly timer), store each chunk's source, version and date, propagate deletions so a withdrawn document disappears from answers, and show the source's date beside the answer so a reader can judge. If you swap embedding models, re-embed everything, because vectors from different models cannot be compared.

Bad chunking

The right document is indexed and the right passage is never retrieved, because the split cut a table in half or separated a clause from the heading that says what it applies to. Anthropic names the problem: in traditional RAG a chunk often loses its context, and its example is a chunk reading 'the company's revenue grew by 3%' that does not say which company or period. Its contextual retrieval method prepends a short, chunk-specific explanation before embedding and indexing, and it reports better retrieval in its own tests. Treat its results as encouraging and not as a forecast for your corpus. The same write-up lists chunk size, boundaries and overlap among the things that affect results, and advises always running evals.

Permissions leaking across users

This is the failure that matters most for a business, and the one a demo never shows. If the index holds everyone's documents and the only thing keeping a junior's question away from the board papers is an instruction in the prompt, then it is a request, and a determined or merely unlucky question will get past it. A model cannot be trusted to withhold what it has been shown. The control has to sit in retrieval, covered in the next section.

Citations that look right and are not

A model asked to 'cite your sources' can produce a plausible reference that does not exist, or a real one that does not say what is claimed. There are two layers of defence. First, never let the model write the pointer. Where the link and title come from your records, the model has nothing to invent: on the marketing platform we built, the link and title of a news item come from the feed, and the model only classifies and summarises, so it has no way to fabricate a source. Second, use the vendor's structured citation features where they exist. Anthropic's documentation says citations are parsed by the API and are 'guaranteed to contain valid pointers to the provided documents', and describes search result blocks that let it cite your own content with the source and title you supplied. A valid pointer is not proof that the passage supports the claim, so keep a reviewer, or an automated check that the quoted text appears in the source, for anything that matters.

Filter before the model sees anything, using the identity of the person asking. The retrieval step knows who is asking and what they are entitled to read, and it returns only chunks from documents they may see. Whatever does not come back cannot be revealed, summarised or leaked by a clever question.

1. 01Store permissions with every chunk: the document, its owner, the groups or roles that may read it, and its sensitivity. Copy them when you index, and update them when the source system changes them.
2. 02Apply the filter inside the search itself, not after it. Retrieving the top results and then removing what the user cannot see can return an empty or distorted answer, and can leak existence through the shape of the result.
3. 03Take identity from your authentication, never from the question. 'I am the finance director' typed into the box is only text.
4. 04Re-check at read time against the source system for sensitive material, because permissions change after indexing.
5. 05Keep a log of who asked, what was retrieved and what was shown, in a store the application cannot edit. When someone asks why the assistant said that, you need an answer.
6. 06Test it with people, not just prompts. Create users with different rights and check each cannot reach what the other can.

The same principle runs through our other work: AI goes through the same door as everyone else. On the solar operating system, the in-app copilot's writes call the same business logic as the REST API, so gates, permissions and the audit log still apply. A retrieval layer should inherit the permissions your existing systems already enforce, not invent a second set. Treat retrieved text as untrusted too: a passage can contain instructions, so label it as untrusted, keep it out of the system prompt and limit what the model can do. Our article on securing AI agents covers that.

Write the questions before you build anything. Anthropic's testing guidance is to define measurable success criteria and build evaluations against them, and a question set is the version of that for document answering. Gather real questions from the people who will ask them, with the correct answer and the passage it comes from, written down by a person who knows.

• Score retrieval and answers separately. Did the right passage come back? If yes and the answer is wrong, the prompt or model is at fault. If no, chunking, embeddings or search is. Mixing them hides which part to fix.
• Include questions with no answer in the documents. The correct behaviour is 'the documents do not say', and that is the one assistants most often get wrong.
• Include exact-match questions (a clause number, a part code) as well as paraphrased ones, to test both halves of a hybrid search.
• Include permission tests: for each of several users, questions whose answers sit only in documents that user must not see. The expected result is nothing, not a polite refusal after retrieval.
• Include freshness tests: change a document, re-index, and ask again. Delete one, and check it no longer appears.
• Include questions that quote the source wrongly, to see whether a check on citations catches them.
• Re-run the set on every change to chunking, embeddings, prompts, model or document set. It is a regression test for answers.

Our estimating assistant is built around a small version of the same habit. One real job was held out of the library entirely and compared with the draft section by section. Every change to the matching or pricing is checked against a quote that was actually sent, rather than against a headline accuracy figure. A concrete, repeatable test beats an impressive-looking number.

The shapes differ more than the totals, and we avoid quoting figures because they depend on your documents, your volumes and the providers' current rates. Estimate from a sample.

• Long context pays on every question, because the documents travel with it. Prompt caching reduces the repeated cost and latency, according to Anthropic's documentation, but cached tokens still count toward the window. Maintenance is low. The expense is the per-question token bill and the ceiling on corpus size.
• RAG moves cost to the build and the upkeep: the pipeline, the index and its hosting, the embedding calls when documents change, evaluation, permissions sync. A single question is cheaper because the prompt is smaller. People underestimate the ongoing work of keeping the index true.
• Fine-tuning has the training runs, the labelled data to prepare and the repeat when facts or the base model change. It is rarely the cheapest route to knowledge and the saving, if any, is on prompt length for a narrow, stable task.

1. 01Write the question set: real questions from the real users, with answers and source passages, including the unanswerable and the off-limits.
2. 02Try the documents in the prompt, structured with sources and a request to quote, on the largest model tier you can afford for the test. See what scores.
3. 03If it passes and the corpus is small and stable, stop. Add caching and a reviewer, and keep the question set for regression.
4. 04If it fails on size, freshness or permissions, build the smallest retrieval: one document source, a hybrid search, and permissions stored with each chunk from day one.
5. 05Score retrieval and answers separately. Fix chunking and search before touching the prompt.
6. 06Add citations the model does not write, and a check that quoted text appears in the source.
7. 07Run permission and freshness tests with real users of different rights. Do not go live until the expected result is nothing.
8. 08Log every question, retrieval and answer. Review the log weekly at first. Widen the corpus only on evidence.
9. 09Consider fine-tuning last, and only for behaviour, never for knowledge.

A caution about our own record. We have not published a case study of a document question-answering system, and we do not claim one. What we can point to is the nearest relevant work: embedding and hybrid matching of past proposals in the estimating assistant (a working demo, not a live product), the source-bound news pipeline on the marketing platform, and the permission-respecting copilot on the solar operating system. Our AI integration page lists retrieval over your own documents among the things we build, where it beats a query. We build on Claude through its public API and are not an Anthropic partner.