RAG (retrieval-augmented generation) is still how most companies bolt private knowledge onto an LLM. The failure mode is familiar: beautiful answers that cite nothing real, or cite the wrong PDF chunk with total confidence. Users trust the tone; legal trusts nothing.
The problem is rarely “embeddings aren't good enough.” It is pipeline sloppiness — bad chunks, no keyword fallback, no citation requirement, no eval set — dressed up as an AI product.
Fixes that move the needle
- Chunk by structure (headings, tickets, functions) — not blind 500-token windows that split a policy in half.
- Hybrid search: keyword + vector. Pure embeddings miss IDs, error codes, SKUs, and anything a user copies verbatim from a log line.
- Force citations: no source, no answer — or answer with “I don't know.” Make “I don't know” a success state, not a failure.
- Eval sets of real questions from support and eng — refreshed monthly from tickets the bot got wrong.
These are not exotic. They are the difference between a demo that impresses a room and a product that survives the first angry Slack thread from someone who knows the doc better than the model.
Pipeline we ship in Forge
Our default RAG stack is boring on purpose:
- Ingest with structure preserved — markdown headings, ticket fields, code fences. Metadata: source, last-updated, access tier.
- Hybrid retrieval: BM25 or equivalent for exact matches, vector for semantic near-misses, rerank the top-k before context injection.
- Prompt contract: answer only from provided sources; quote or link; refuse if sources conflict or are empty.
- Log retrieval hits per query so you can debug “why did it cite the 2022 policy?” without guessing.
We skip the vector-db beauty contest unless latency or scale forces it. A well-chunked index on Postgres with pgvector has shipped more than one production workload without a dedicated search vendor.
Evals that catch lying
RAG evals need three checks, not one “sounds right” score:
- Retrieval recall: did the right chunk appear in top-k? Fix chunking and search before blaming the model.
- Citation accuracy: does the cited span actually support the claim? Automated string overlap plus spot-check.
- Abstention: on questions with no good source, does the model refuse instead of confabulate?
Run these in CI alongside your prompt evals. A retrieval regression is silent until a customer notices — which is the worst time to find out.
When RAG is the wrong tool
If the truth changes every hour — inventory, pricing, account status — you need tools that fetch live data, not a nightly index job pretending it is current. If the agent needs to act — file a ticket, update a record, send an approval — you need workflows and Embed integration, not a bigger vector DB.
RAG answers “what does our documentation say?” It does not answer “what is true right now?” Know which question you are building for.