RAG (Retrieval Augmented Generation) is the technique that allows a language model to answer using your organization’s documentation instead of only what it learned during training. It retrieves the relevant passages from your sources, adds them to the query and generates an answer that can be verified against the original document.
What RAG is and why it matters in an enterprise project
Most generative AI projects in business hit the same wall: the model writes well, but it does not know your business. It does not know what your quality procedure says, which clause is in your framework agreement, or which was the latest version of the clinical protocol. And when it does not know, it sometimes makes it up.
RAG stands for Retrieval Augmented Generation. The technique was formally documented in 2020 in a paper by Patrick Lewis and his team presented at NeurIPS, and it consists of combining the model’s parametric memory (what it learned during training) with an external non-parametric memory: an index of your own documents that the system queries at the moment of answering.
The practical consequence is what matters to a management committee. Each answer can cite its source, information is updated by changing the document instead of retraining the model, and sensitive knowledge stays in your infrastructure.
How a RAG system works under the hood
A RAG system is usually described in two phases, retrieval and generation. In a real project it is worth looking at it in four stages, because 90% of failures appear in the first two and almost never in the last one.
Document ingestion and chunking
Documents are processed and split into manageable chunks. This stage looks trivial and is the one that most determines the final result: a PDF with tables, a contract with annexes or a scanned presentation requires structured extraction before chunking. Semantic chunking, which respects paragraph, section and table boundaries instead of cutting every thousand characters, is the difference between a system that answers and one that returns mangled fragments.
Vector indexing and hybrid search
Each chunk is converted into a numerical vector using an embedding model and stored in an index that allows searching by semantic proximity. Purely vector search fails with proper names, product codes and exact references, so in an enterprise environment it is combined with classic lexical search. That combination is hybrid search, and it is practically mandatory when your documentation is full of identifiers.
Reranking and context building
The search returns more candidates than fit in the model’s context, and they are not ordered by real usefulness. A reranker re-evaluates them and keeps the best ones. Then you have to decide how many chunks go in, in what order and with what metadata, because models pay less attention to what sits in the middle of a long context. That design work is what is known as context engineering.
Generation with source citation
The model receives the augmented query and generates the answer. If the index keeps the title, URL and section of each chunk, the answer can cite where each statement comes from. Without that traceability the system is hard to audit, and in regulated sectors that makes it unviable.
Enterprise RAG architecture: the components you do need
A RAG proof of concept can be put together in an afternoon. An architecture that holds up in production has five pieces that the demo did not need.
- The first is an ingestion layer with scheduled pipelines: documents change, and an index that is not updated starts to lie.
- The second is the search engine and vector index itself. The Azure AI Search documentation sets out two models for this: the classic pattern, with hybrid search and semantic reranking, and agentic retrieval, where a model breaks the complex query down into subqueries that run in parallel and are reranked before being merged.
- The third piece is document-level access control, which we cover further down because it is usually the one that sinks projects.
- The fourth is the evaluation layer: without a set of questions with expected answers, there is no way of knowing whether a change improves or worsens the system. And the fifth is observability, so that you can reconstruct which chunks were retrieved when someone reports an incorrect answer.
The real problems of RAG in production
Microsoft acknowledges in its own documentation that RAG implementations face significant challenges, and this matches what we see in the field. These are the six failure modes that come up again and again.
Poor chunking ruins retrieval
If chunks are too large, they saturate the context and dilute the signal. If they are too small, they lose the context that made the sentence understandable. And if chunking ignores structure, a dosage table is split in half and the system answers with half a table. It is the most common failure and the easiest to fix.
Retrieval that does not find what does exist
The user asks about “the batch release procedure” and the document calls it “SOP-047”. Vector search does not bridge that gap on its own. This is where hybrid search and a corporate synonym dictionary stop being refinements and become requirements.
Permissions: the problem that shows up on demo day
If the index contains all the documentation and the system does not filter by identity, any employee can extract payroll, case file or negotiation information through the chat. Control has to be applied at retrieval, filtering which chunks are visible to whom before they reach the model. Adding it later forces you to rebuild the architecture, and it is the most frequent reason a pilot does not reach production.
Prompt injection from the documents themselves
Retrieved content enters the model’s context, and if a document contains text written to manipulate the system, the model may obey it. In an internal repository the risk is low. When emails, tickets or documents coming from outside are indexed, it stops being low.
The index that goes stale
A RAG system with a six-month-old index gives outdated answers with the same confidence as if they were current, and that is worse than not answering. Incremental updating and deletion of retired documents have to be solved from the design stage.
Evaluating quality is harder than it looks
Asking ten users whether they like it is not an evaluation. You need to separate two things: whether the system retrieved the right chunks and whether it generated an answer faithful to those chunks. A retrieval failure and a hallucination are fixed in different ways, and without measuring them separately you are optimizing blind.
Techniques to improve a RAG system that is not working
When a RAG system answers badly, the usual reflex is to change the model. Almost always the problem lies in retrieval, and there are four levers to pull before touching the model.
- Hybrid search combines the vector with lexical matching and solves the identifier problem.
- Reranking with a cross-encoder model reorders the candidates and usually gives the biggest improvement per unit of effort.
- RAG Fusion generates several reformulations of the original question, retrieves for each one and merges the results, which widens coverage when the user’s query is ambiguous.
- And agentic retrieval takes that idea further: a model plans how to break down the question, launches the subqueries in parallel and returns a structured set, also making use of the conversation history.
None of the four requires retraining anything. All of them act on the half of the system where the failures are.
RAG or fine-tuning: decision table
They are techniques that solve different problems and are often combined. This is the quick way to decide.
If your problem is that the model does not know your data, the answer is RAG. If your problem is that it answers in a register that does not fit your organization, it is fine-tuning. If it is both, they are combined.
How much a RAG project costs and when it pays off
The cost of a RAG system is split across three items: preparing the document base, which is usually the most underestimated; the index and compute infrastructure, which scales with the volume of documents and queries; and the model’s cost per query, which grows with the size of the context you send it.
- It pays off when there is a critical mass of documentation that people consult daily and take a long time to find, when that documentation changes frequently, and when there is a traceability requirement that forces you to justify every answer.
- It pays off less if your document base is small and stable, or if your users’ real questions can be solved with a well-configured search engine. It is worth ruling it out before investing.
Use cases where RAG delivers measurable value
In customer service and technical support, an assistant that consults the official product documentation reduces resolution time and avoids contradictory answers between agents. In internal knowledge management, the value shows up in onboarding new people and in looking up policies and procedures, something that today is solved by asking a colleague.
In the legal field, RAG speeds up searching case law, legislation and the organization’s own contracts, with source citation as a non-negotiable requirement. In healthcare and pharma, it makes it possible to consult clinical and regulatory documentation while keeping access control and an audit trail of every query. And in software development, an assistant over the documentation of internal systems reduces the time a team loses reconstructing how what already exists works.
The common pattern is the same: a lot of documentation, frequent queries and a need to verify where the answer comes from.
How we approach RAG projects at BertIA
At BertIA we start in the least glamorous place: the document base. Before choosing a model or an index, we assess what documentation exists, what state it is in, who can see what and which real questions the business needs to answer. Without that resolved, any architecture fails in the same place.
We work on Databricks and Microsoft Fabric, and we design access control and data governance from day one instead of adding them once the pilot is already working. We also define the evaluation set before building, so that we can demonstrate whether the system improves. You can see the full scope of our artificial intelligence solutions to understand how a RAG system fits into a broader data architecture.
Conclusión
A well-scoped RAG project turns static documentation into knowledge that can be queried, with the source of each answer in plain sight. The ones that fail almost never do so because of the model: they fail because of careless chunking, retrieval that does not find what exists, permissions added late or the lack of a way to measure whether anything improves.
If your organization manages large volumes of technical, regulatory or product documentation and you want to assess whether RAG solves a real problem before investing in it, you can contact our team to review your specific case.
