Ask a public AI assistant about your company’s return policy for business customers and you will get a confident, well-written answer. It will also be made up. The model has never seen your terms and conditions, your price list or the email from last Tuesday that changed the delivery times.
Most companies hit this wall in the first week of trying AI. The usual fix is not training a new model. It is a technique called retrieval-augmented generation, or RAG: before the model answers, the system looks up the relevant passages in your own documents and hands them over. The model answers from those passages and shows where the answer came from.
This post explains how RAG works, what makes it good or bad, and when it is the right tool.
Why the model doesn’t know your company
A large language model learns from a huge amount of public text, up to a cut-off date. Three things follow from that:
- It has never seen your internal documents, contracts, tickets or wiki.
- Its knowledge stops at its training date. Last month’s prices are not in there.
- It doesn’t remember earlier conversations unless the app sends that history again.
You could paste your whole handbook into every question. For one short document, that works. For hundreds of documents it doesn’t. Everything the model reads costs money and time, every model has a limit on how much it can read at once, and answers get less reliable when the useful paragraph is buried in a long input. A 2023 study led by Stanford researchers found that models use information at the start and end of a long input much better than information in the middle, and named the effect “lost in the middle”. Our post on tokens and context windows explains these limits.
RAG gives the model only what matters for this one question.
How RAG works, step by step
The name comes from a 2020 research paper by a team at Facebook AI Research, now part of Meta. The idea is simple: retrieve first, then generate.
Retrieval-augmented generation
- Question In the user’s own words
- Search Find passages that match the meaning
- Best passages Re-ranked and filtered by permissions
- Model writes Answers only from those passages
- Answer with sources Links to the paragraphs it used
1. Prepare the documents
Before anyone asks anything, the system collects the documents from where they live (SharePoint, Google Drive, a wiki, the helpdesk), cleans them and splits them into chunks of a few paragraphs. A good chunk makes sense on its own: one section of a policy, one FAQ answer, one clause of a contract. Each chunk keeps its metadata: document title, date, owner and who is allowed to see it.
2. Turn meaning into numbers
Each chunk goes through an embedding model, which turns text into a long list of numbers that represents its meaning. Texts that mean similar things get similar numbers. That is why the question “how many days off do I get?” can find a paragraph titled “Annual leave entitlement”, even though they share no words. The numbers are stored in a search index, often called a vector database.
3. Search, then re-rank
When a question arrives, it is turned into numbers the same way, and the index returns the closest chunks. Good systems combine this search by meaning with ordinary keyword search, because product codes, names and contract numbers need exact matches. A second model then re-ranks the 20–50 candidates and keeps the handful that best answer the question.
4. Write the answer, with sources
The model receives your instructions, the selected passages and the question. The instructions say, in effect: answer only from these passages, name the source of each statement, and say so if the answer isn’t there. The user gets an answer with links to the paragraphs behind it, so they can check.
What a RAG system is made of
- Chat or search box In the intranet, the helpdesk or Teams
- Answer model Instructions + passages + question
- Retrieval Search by meaning and keywords, re-ranking, permissions
- Index Chunks, embeddings, metadata
- Your sources Drive, SharePoint, wiki, tickets, PDFs
Here is what that looks like for an employee of a wholesale distributor:
- You: Can we give a new distributor 60-day payment terms?
- Tool call: search(“payment terms for new distributors”)
- Result: Credit policy v3, section 4.2 · Distributor onboarding checklist, page 2
- AI: Not without approval. New distributors start on 30 days. Longer terms need a credit check and sign-off from the finance director (Credit policy v3, section 4.2). The form to use is in the onboarding checklist (page 2).
Keyword search or RAG?
Finding an answer in company documents
Keyword search
- Guess the exact words the document uses
- Get 40 results, half of them outdated
- Open the PDFs one by one
- Find the right paragraph yourself
RAG assistant
- Ask in your own words
- It reads the best passages for you
- One answer, with links to the source
- It says when the documents don’t cover it
RAG doesn’t replace your documents or your search. It sits on top of them. If the documents are wrong, it will give you a wrong answer with a citation.
What makes RAG good or bad
Most RAG projects that disappoint fail at retrieval, not at the model. If the right paragraph never reaches the model, no model can answer well.
Document quality
Three versions of the same policy, scanned PDFs with poor text recognition, key facts hidden in tables or images, and pages that just say “see attachment”: these cause more wrong answers than anything else. Before you index everything, pick the sources someone actually maintains and remove the old versions.
Permissions
An assistant must not show a warehouse worker the salary bands just because they sit in the same SharePoint. The system has to filter search results by what the person asking is allowed to see, at search time, before the model reads anything. Ideally it reuses the permissions you already have in SharePoint or Drive instead of keeping a second set.
Freshness
When a document changes, the index has to change with it. Plan for automatic re-indexing, and show the document date next to every source so people can spot an outdated answer.
Evaluation
Collect 50–100 real questions with known correct answers before you build. Then check two things separately: did the search find the right passages, and did the model answer correctly from them? Run the same set after every change. Without it, you are tuning by feel. Our post on why AI makes things up covers more ways to keep answers grounded.
RAG, long context or fine-tuning?
- Paste it into the prompt when the material is short and specific to the task: one contract to review, one report to summarise.
- Use RAG when you have many documents, they change, different people may see different things, and you need sources.
- Fine-tuning changes how a model behaves: its style, its format, a narrow task. It is a poor way to teach facts, because the model can’t cite them and they go stale. We compare all three in prompting, RAG or fine-tuning.
One more distinction. RAG is for text. If the question is “what were our sales in Bratislava last quarter?”, the answer sits in a database, not a document, and searching text chunks is the wrong tool. There, the model should query the database through a tool. That is how Charlie, the assistant we built for Shovels, answers questions about building permits from live records.
Where RAG works well
- Internal knowledge assistant: HR policies, IT how-tos, process descriptions; the questions people now ask a colleague.
- Support answers: draft replies for support agents from the help centre and past resolved tickets, with the source shown so the agent can check before sending.
- Policy and contract lookup: “what is our liability cap with this supplier?” across hundreds of contracts.
- Technical documentation: manuals and service notes for technicians who need one answer on site, not a 300-page PDF.
RAG is less useful when you have 20 FAQ answers (put them in the prompt, or on a good FAQ page), when the documents are a mess nobody owns, or when the answer needs judgement rather than lookup.
What a first version takes
A focused assistant over one well-kept set of documents is usually a small project. Most of the work isn’t in the model. It is in the plumbing around it: connecting the sources, splitting documents sensibly, keeping permissions right, re-indexing when something changes, and testing against real questions.
As a rough guide from our own pricing: an assistant connected to one or two sources falls in our “one process” range of €5–10k. An assistant built into a portal or an internal tool, with several sources and permissions per user, falls in the €10–30k range. Our projects have reached a first working version in 6 weeks to 3 months.
Running costs come from three places: the model reading passages and writing an answer for each question, turning new and changed documents into embeddings, and hosting the index. The first one grows with the number of questions, so estimate it from how many questions you expect per month, not from a demo.
The cost people forget is ownership. Someone in the business has to keep the sources current and look at the questions the assistant couldn’t answer. Those unanswered questions are the most useful feedback you will get: they show you which documents are missing.
How to start
- Pick one group of users and one set of documents, for example the support team and the help centre.
- Collect 50–100 real questions with the right answers. This is your test set.
- Clean the sources. Remove old versions and name an owner for each source.
- Decide permissions, reusing what your existing systems already know about who may see what.
- Build a pilot that answers with sources and says “I don’t know” when nothing matches.
- Review after four weeks: what people asked, what failed, what to add next.
If your documents can’t leave your own environment, the same setup can run on self-hosted models; see private AI systems. And if you want a second opinion on whether RAG fits your case, get in touch.
