Retrieval-augmented generation has a slow-poisoning problem. The system that impressed the room in month one is quietly worse in month six, and nobody notices until a customer forwards a support ticket with the wrong answer attached.
Three usual suspects. All fixable. All routinely ignored.
1. Your embedding model updated
OpenAI, Voyage, Cohere, Google — all ship new embedding models on a rolling basis. Sometimes the API endpoint stays the same and the vectors change. Sometimes a deprecated model quietly falls out of parity with its successor. A vector produced by yesterday's model and a vector produced by today's model are not comparable, even when the two models share a family name.
Fix: store the embedding model identifier alongside every vector. Refuse to serve a query whose model identifier doesn't match the corpus. Not "log a warning" — refuse. A silently-wrong retrieval is worse than a hard fail.
2. Your data drifted, your vectors didn't
Your knowledge base grew. New product names, new acronyms, new customer verticals. The old vectors know none of them. Users search for the new terms and get the closest cosine match — which is not the same as the right match.
Fix: instrument top-k retrieval with a "no result crossed the confidence floor" counter. When that count spikes on a specific query cluster, it's telling you something entered the vocabulary that isn't in the index.
3. Nobody re-indexes because re-indexing is scary
Full re-embedding of a large corpus takes hours and a real bill. Nobody wants to be the person who kicks it off, so nobody does, so drift compounds.
Fix: put a re-index budget in the CDK. A scheduled job, once a quarter, embedding the whole corpus with the current model against a versioned index alias. Cut over when precision@k on your eval set is better than the live index. Roll back if it isn't. This is boring infrastructure and it is what keeps the answer honest.
Eval the retrieval, not just the LLM
Most teams eval the final generation and shrug at the retrieval. That is backwards. The retrieval is where the accuracy budget is spent. Build a small labelled set — a hundred queries, human-graded top-3 — and run precision@k every week. When it drops, you'll know before your users do.
There is a research literature on this now: HyDE, contextual retrieval, matryoshka embeddings, re-ranking cascades. Read it, but treat it as a menu. Your product's constraints — latency, index size, cost per query — decide the dish.