Your RAG Regressions Are Found by Users, Not by CI
The bug report a retrieval system earns is never an error code. It is a Tuesday email saying the permits portal used to quote the 2024 fee schedule and this week it quotes 2023. The writer is right, nothing crashed, no dashboard turned red, and the change responsible merged a week earlier with a clean diff and an approving review. If an email shaped like that has ever carried your name in the to field, this post is the argument for what should have caught it first, an eval suite that runs in CI.
The claim in one sentence: most retrieval regressions are cheap to catch and expensive to find, and a few hundred golden questions running in CI catch more of them than any infrastructure upgrade you could buy this quarter. The suite is unglamorous software work, a versioned data file, a script, and one pipeline stage that turns red before the merge. Order matters more than tooling. Evals first, then scaling, is the sequence that does not spend a quarter buying capacity for answers nobody has audited.
A retrieval regression is a silent swap of the quoted sentence
Nothing in a retrieval stack raises a flag when it misbehaves, because each stage succeeds at its own job. The chunker still chunks. The embedder still embeds. The store still answers pings. What changed is which five documents float to the top, and the generated answer follows them, so the system returns 200 with 2023's amendment in hand instead of 2024's schedule. Both documents contain the word renewal, which is why the grep you ran through the staging logs finds nothing.
The usual defenses fail quietly as well. The spot check covers five favorite queries, and favorites age: they exercise what the team already knows, which by definition is not where the next miss lives. The staging smoke test stays green through every one of these changes, because a retrieval regression is a correctness failure wearing the clothes of a successful request. Neither check asks the only question that matters, which is whether the same question received the same grounding this week as last week.
And the stack has enough moving parts to make silent drift an ordinary event rather than an accident. Chunk size, embedding model, index rebuild, reranker threshold, the wording of the system prompt: each is a one-line diff, each reshapes what the next stage sees, and a sensible cleanup in one layer is a regression in the answers. By late 2025 this is the standard shape of a RAG outage, and the fix lives in the same place any other regression's fix lives, in CI.
Golden questions come from the logs, not from meetings
Our own first eval set failed in the standard way. Twenty questions, written in a conference room by the people who built the system, and it passed for a quarter while the misses kept arriving, because the authors never ask the odd, half-typed questions users ask on a Friday afternoon. The correction is humbling in how small it is. The questions come from the product's own query logs, sampled across a quarter, typos included, abbreviations included, and the questions people phrase as apologies included.
What comes out of that sampling is a versioned file that lives in the repository, reviewed and edited like the code it protects:
- question: "What is the fee to renew a commercial permit?"
expected_source: "fee-schedule-2024.pdf"
answer_contains: "renewal"
- question: "Why did the council reject the harbor variance?"
expect_refusal: true
- question: "fee renewal comercial permit"
expected_source: "fee-schedule-2024.pdf"
Three fields carry the weight. expected_source names the document that must appear in the top five hits, which pins retrieval itself. answer_contains is the substring the generated answer must include, which pins generation without grading essays. expect_refusal marks the questions the corpus cannot answer and the system must decline, because out-of-scope requests are where a RAG system invents its most fluent nonsense. The misspelling in the last row is not decoration: plenty of real traffic is phrased worse than that.
A few hundred is the working number, neither twenty nor twenty thousand. Enough rows that a bad change trips several of them at once, few enough that a human can reread every failure the job reports. Two habits keep the file honest: it grows by one row whenever a user reports a miss, so every complaint becomes a regression test, and old rows are pruned when the documents they name are retired, so the suite does not spend its runtime testing history.
Where the job runs matters less than one detail, and the detail is a frozen corpus. Retrieving against live data confuses code regressions with content drift, and a failing merge you cannot blame on a diff is a failing merge nobody trusts twice. Pin a small snapshot of the documents, versioned with the case file, so every failure attributes to something a commit actually changed.
Three assertions carry most of the load
The retrieval hit is the assertion that catches the most, and it runs before a single token of generation is billed. Does the document that should ground this answer appear in the top five? A chunk-size change, an embedding refresh, or an index rebuild, the three changes that broke the fee schedule above, all show up here in seconds:
def test_golden_case(case, rag):
hits = rag.search(case["question"], k=5)
sources = [hit.source for hit in hits]
assert case["expected_source"] in sources, case["question"]
if case.get("answer_contains"):
answer = rag.answer(case["question"], hits)
assert case["answer_contains"] in answer, case["question"]
That is most of the machinery. A pytest fixture reads the YAML file and parametrizes each row into a test, the CI job fails the merge when any assertion fires, and a case with structured output checks the JSON shape against a schema before anything else touches the payload.
Containment covers the facts with a canonical wording; an LLM-as-judge covers the open-ended remainder, graded against a written rubric rather than a general impression. Two cautions come from practice. Pin the judge model like a dependency: we watched an unpinned judge upgrade itself mid-quarter and move scores a full band with no change to the system under test, which cost an afternoon of forensics. And reread a sample of the judge's verdicts by hand for a while, because a judge that grades confidently and wrongly is just the regression from paragraph one wearing a new coat.
Off-the-shelf suites such as promptfoo and Ragas package variants of these assertions plus the judge plumbing, and they are reasonable choices. The asset is not the tool, though. The asset is the case file that encodes what your users actually ask, and it transfers into any runner you ever adopt.
Where a CI eval suite is blind
Four honest limits, each with its workaround.
The suite certifies consistency against yesterday's answers, not truth against the world. A change that makes the system confidently wrong in a new way passes green. The workaround is a small recurring audit: a handful of cases each month, checked by a person against the source documents the citations point to, on top of the pruning habit that keeps the case file tied to documents that still exist.
A few hundred cases leave slices uncovered. Dialects, industry jargon, questions ten words long: whatever the log sample under-weights, the suite under-watches. The defense is not more rows at random but revisiting the stratified sampling habit from the build section whenever a new market or a new document family goes live.
Latency and cost are budgets the suite never sees. An eval pass over a few hundred cases will not notice that retrieval takes four times longer than it did before a change, because nothing in an assertion measures duration. Load tests and cost reports stay on the calendar, unglamorous as ever.
And the judge caveat closes the list: scores move when the judge changes, so the judge version stays pinned, reviewed, and upgraded on purpose rather than by accident. It is one more dependency in the lock file, treated exactly like one.
The decision to take away fits one sentence: before the next retrieval or model change lands, spend a week building the golden-question file rather than a quarter buying capacity to serve unaudited answers. Two hundred questions from the logs, three assertions, one CI stage that fails the merge when a hit drops. After that, the scaling conversation has evidence a skeptic can rerun, and the Tuesday email meets a competitor: a pipeline stage that says the word failed before the merge goes through.
AngoraTek builds retrieval systems for small teams, and the eval suite we install first usually predates the scale-up it guards. If your dashboards are green and your users disagree, bring us the Tuesday email.Thanks for reading this far. If this sounds like the problem you're wrestling with, tell us about it. The team reads every message.
Start a conversation