Over the last 18 months, almost every discovery meeting we start with an SME ends with the same sentence: "we want a ChatGPT, but with our documents". The market’s answer to this is always the same — RAG (Retrieval-Augmented Generation). And sometimes it’s the right answer. Other times, it will take you 6 months to find out it wasn’t.
This post captures the framework we use at K2 Labs to decide, in less than a week of work, whether a RAG project will work for a specific client. It isn’t definitive. But it has saved us from three commitments that would have been catastrophic.
What is RAG, in plain terms?
RAG is the pattern "find relevant documents, feed those documents to the model, and have it answer using that information". It’s the difference between asking a model something cold and giving it top-k fragments of context before the question.
Mechanically, it’s three pieces:
- Indexing: your documents are split into chunks, vectorised with embeddings and stored in a vector store.
- Retrieval: given a question, you fetch the most similar chunks.
- Generation: you pass the question + the chunks to the LLM and it answers.
In 2026, wiring this up with LangChain or llama-index is a Friday afternoon. What’s hard is making it answer well.
When it DOES make sense
RAG’s sweet spot is fairly specific. It works very well when:
- The user asks factual questions with a canonical answer somewhere in a document.
- Your corpus is relatively stable and curated — manuals, policies, internal knowledge bases.
- There is enough volume of usage to justify maintaining the system.
- You accept that the answer cites sources and that the user can read them to validate.
A real case: an industrial distributor with 4,000 technical datasheets in PDF. Call-centre technicians spent 40% of their time searching for specifications. RAG cut that figure to 8%, with a clear ROI in 3 months. Here, every box was ticked.
When it does NOT
And here comes the nuance that’s rarely discussed. RAG is a bad idea when:
- Answers require reasoning across multiple documents at once that contradict each other.
- The content changes constantly and indexing has latency.
- The questions are procedural or transactional — that’s an API, not a semantic search.
- The corpus is small (under 100 docs). A good prompt with the docs in context performs just as well or better.
If the problem is solved by full-text search + an LLM at the end, you don’t need RAG. You need Elasticsearch and a prompt.
The decision framework in 5 questions
In the first meeting, we run this battery. If three or more answers are "no", we recommend a different approach.
- Does the user’s question have a concrete answer within the corpus?
- Does the corpus have more than 500 meaningful documents?
- Is the content updated weekly or slower?
- Is there a human baseline to measure accuracy against?
- Do users accept an answer with citations?
Closing
RAG is not a silver bullet, but it’s not a failure either. It’s a specific architecture for a specific problem. The hard part of a consultant’s job in 2026 is no longer building the pipeline — it’s having the honest conversation about whether your problem fits.
If you’re considering a project like this and want a second opinion before committing, write to us. One hour of discovery can save you a semester.