Prompt Regression Tester

Compares a trusted prompt with a proposed change over the same test suite, three runs per case, and calls a regression only when the baseline holds and the candidate keeps failing. By Benjamin Nweke Project page: https://towardsdatascience.com/shipai/projects/prompt-regression-tester ShipAI is a Towards Data Science project. You can learn more about it here: https://towardsdatascience.com/shipai/about Key takeaways. 1. Each case runs three times per prompt, so a regression means a stable baseline pass and a stable candidate fail. 2. Hard checks guard dates, numbers and review states; an LLM judge with a rubric scores meaning, and disagreements stay visible. 3. On an EvidenceDesk suite, a candidate that swapped dates for “earlier” and “later” failed the versioned case 3 of 3. Chapters: 0:00 Why prompt changes deceive 0:52 An EvidenceDesk-style suite 2:07 Baseline versus candidate 3:16 Repeated runs and split checks 4:42 Under the hood 5:25 One confirmed regression 6:19 When dates are evidence 8:10 When the judge disagrees