"Retrieve documents, stuff them in a prompt, generate an answer." That's the three-step description of RAG you'll find everywhere - and it's also why so many RAG pilots look identical on day one and diverge wildly by month three. One system answers reliably at scale; another quietly hallucinates, leaks documents it shouldn't, or degrades the moment real users start asking real questions. The three-step version describes what happens. It says nothing about where things break.
After building and reviewing RAG systems across different enterprise contexts, I've stopped trying to diagnose failures at the "retrieve - prompt - generate" level - it's too coarse to be useful. Every recurring failure I've seen traces back to one of six specific layers, each with its own failure mode, its own fix, and its own owner. This isn't a theoretical framework imposed on the problem; it's the map that emerged from watching the same things go wrong, repeatedly, across systems that all started from that same three-step description:
Click to enlarge
Layer 1 - Ingestion & Preprocessing
This is where most RAG quality problems actually originate, and it's consistently underfunded in project timelines. Connectors pull documents from wherever they live, parsers extract clean text, and chunking splits everything into retrievable pieces with metadata attached. If your chunking strategy is naive - fixed character counts with no awareness of document structure - everything downstream inherits that damage.
Clean connectors, structured parsers, and consistent chunking produce high-quality retrievable pieces with accurate metadata downstream.
You'll spend the first sprint on PDFs alone. Tables split across pages, headers without their body, scanned images masquerading as text. The chunking strategy from week one gets replaced by week four.
Layer 2 - Embedding & Indexing
Each chunk becomes a vector, stored in an index supporting both semantic and keyword search, with metadata filters for access control. The access control piece is routinely underestimated - if your retrieval doesn't respect document permissions, you have a data leak waiting to happen, not just a quality issue.
Layer 3 - Retrieval
The query gets rewritten and matched against the index, with reranking to push the most relevant chunks to the top. This is the layer the three-step definition points to as "RAG" - but it's the third of six, and its output quality is bounded entirely by what Layers 1-2 gave it to work with.
Semantic search surfaces the most relevant content for any query by matching meaning, not just keywords.
The first time a user types a product SKU and gets back a marketing essay, you add keyword search. Hybrid retrieval isn't an optimization - it's the minimum viable strategy for any corpus that mixes prose with identifiers.
Layer 4 - Orchestration
Retrieved chunks get assembled into a clean context window: deduplicated, compressed, and screened by guardrails before reaching the model. This is also where prompt injection defenses and PII redaction belong - not as an afterthought, but as a structural part of the pipeline.
Layer 5 - Generation
The model produces an answer grounded in the assembled context, with prompt instructions requiring citations back to source chunks. Citation isn't a UX nicety - it's how you make the system's outputs auditable.
Layer 6 - Memory & Feedback
Conversation history and user feedback loop back into Layers 1-4, continuously tuning retrieval quality based on real usage. This is the layer that turns a static system into one that gets better over time - and the one proof-of-concepts consistently omit.
Key insight for this series: higher-maturity RAG patterns - Advanced, Modular, Graph, Agentic - don't replace this architecture. They add components inside these six layers. Every later addition has an obvious home once you understand the foundation.
RAG Architecture Series - 6 Parts