A Prompt Change Is a Code Review That Starts With a Failing Case
The prompt that changed your product's behavior on a Saturday was not written on a Saturday. It was one clause added to a system prompt eleven days earlier, merged by the person who wrote it, because prompt edits ride along with deploys and nobody's review is required. A sentence of prose that reaches every request a product makes deserves the caution a CSV parser gets. It rarely gets it.
So we review prompts like code, and the review starts from a failing case instead of from a diff. The reasoning is mechanical: prose hides its consequences, a shared system prompt reaches every case the product answers, and no type system exists to warn a reviewer about the clauses that will bend behavior somewhere else. Diff-first review reads the change. Failing-case-first review reads what the change does, and the difference is where the regressions live.
One reworded clause bends behavior far from it
The hard fact about prompts is what they lack: a compiler. Change an instruction in one corner of a shared system prompt and behavior moves in corners the diff never mentions. We watched a team add a plain clause about keeping answers short, and a week later a JSON export began losing keys, because the model had started abbreviating field names to honor its new brevity rule. Nothing in the diff pointed at the export. Nothing could have.
The conflict to name is iteration speed against blast radius. Prompt work feels like writing, and writing rewards fast changes, so every gate you put in front of a prompt edit feels pedantic on a Tuesday afternoon. The blast-radius half of the argument only shows up on a Saturday, and by then the gate is history. A review process that starts from failing cases is how both halves survive: speed stays high because review asks one question, and the blast radius shrinks because the question is the one that matters. That question is simply: which case gets fixed, and which case must not move?
The failing case file opens the review
Before the reviewer sees a diff, the change carries a case with it: an input, the output the old prompt produced, the output the new prompt should produce. The format can be plain:
case_id: json-export-keys
prompt_version_before: git:8a41f2c
prompt_version_after: git:d0e77b1
input: |
Summarize this shipment manifest and return strict JSON.
Manifest: [attached fixture manifest-407]
expected_output: |
{ "shipment_id": "MF-407", "packages": 18, "weight_kg": 312.4 }
observed_output_before: |
{ "id": "MF-407", "pkgs": 18, "kg": 312.4 }
must_not_change:
- refusals on policy questions
- tone of the French-language answers
The same format, accumulated incident by incident, becomes the golden set: the cases the change must not break, run automatically on every proposal. Naming matters more than it looks here: a case named for the symptom it breaks gets found again months later, while case_12 turns into archaeology. The golden set exists because the failing-case file kept growing. Each real regression that reached production entered it as a case, and by the time the file held a few dozen entries, running it before merge stopped most of the debates a prompt review otherwise collapses into. The step after the failing case is therefore not process. It is the same file, matured.
The checklist, and why it stays small
The checklist a reviewer runs is six lines, and six is deliberate, because a long checklist teaches people to skim:
- One intent per change. A diff that rewords and adds belongs in two changes.
- The named failing case is linked, and its case runs green on the proposal before review starts.
- The golden set runs and is quoted in the change description, including its misses.
- A pure rewording ships with an output diff, not just a text diff, so the reviewer compares behaviors rather than sentences.
- The previous prompt version survives, tagged, so rollback is a revert and not an archaeology.
- Anything the evals cannot reach gets said out loud in the description, as a risk, in a sentence.
The judge that grades the judge
Most prompt changes get graded by an LLM reading outputs against a rubric, and a grading rubric is itself a prompt. That symmetry is the honest limit of the whole method: the grader drifts when its underlying model updates, it can be gamed by outputs tuned to sound correct, and its reliability is exactly as reviewed as you make it. We version the grader prompt in the same repository as the prompts it judges, and we review changes to it under the same rule, so a grader change needs a failing case too. Usually that failing case is an output the old grader waved through that a human reads and rejects. Once a week or so, a person hand-checks a sample of graded outputs against the rubric, and the size of the sample is a decision made in the open rather than a comfort.
What review still misses, and what we do about it
Evals encode what you anticipated, and the wild does not. Real regressions arrive in shapes no case in the file predicted, and the answer is not a longer checklist. The answer is a door: anything the wild sends becomes a new failing case in the file within the day, with the person who found it named as the case's owner. The file grows toward the product instead of toward the author's imagination.
Our own slow lesson was the eleven-day clause from the opening, and it was ours. The clause merged without a review because the change looked like prose, and the export bug surfaced later from a different team entirely. The correction that stuck was not a rule but a habit: review comments quote expected outputs, not prompt text. A sentence about sentences invites editing. An output diff invites a decision.
If your team is choosing whether to adopt any of this, the cheap version is one step: no prompt change merges without a failing case attached. The golden set, the grader review, and the checklist all follow from the file that step creates, in the order they are needed.
AngoraTek runs prompt reviews like this for features our clients ship to their own customers. If your system prompt has grown past the point where an edit is an edit, show it to us and we will open the first failing case together.Thanks for reading this far. If this sounds like the problem you're wrestling with, tell us about it. The team reads every message.
Start a conversation