Pre-deploy eval harness
Diffscore runs your suite before and after the change, then tells you whether the difference is real — a paired significance test instead of a pass count, and a judge calibrated against labels you wrote yourself.
pip install diffscore- MIT licensed
- Bring your own key
- Runs in CI
- Caches every call
Significant regression on 4 casesp = 0.02
McNemar’s paired test over 60 shared cases. Exits 1, so CI fails.
added "be concise" to the system prompt| Case | Grader | Baseline | Current | Δ | Verdict | |
|---|---|---|---|---|---|---|
| refund-window-eu Dropped the 14-day EU refund window entirely. | judge:faithfulness | 1.000 | 0.000 | −1.000 | regressed | |
| refund-policy-us States the policy exists but omits the 30-day limit. | judge:faithfulness | 1.000 | 0.250 | −0.750 | regressed | |
| escalation-tier2 Required string "human agent" no longer present. | contains | 1.000 | 0.000 | −1.000 | regressed | |
| tone-frustrated-user Terse to the point of reading as dismissive. | judge:helpfulness | 0.850 | 0.400 | −0.450 | regressed | |
| order-status-json Now emits valid JSON; the preamble that broke parsing is gone. | json_schema | 0.000 | 1.000 | +1.000 | improved | |
| greeting-brevity Shorter output, 1.1s under the ceiling. | max_latency_ms | 0.620 | 0.910 | +0.290 | improved | |
| 54 unchanged cases hidden. --show all to list them. | ||||||
The whole workflow
Three steps, under five minutes.
There is no fourth step, and no dashboard you have to live in. If you cannot get from your own prompts to a verdict in one sitting with your own API key, the tool has failed.
- 01
Define a suite.
Inputs plus what good looks like. A YAML file in your repo next to the prompt it tests, or a CSV upload if you would rather not write YAML.
suites/support-tone.yamlname: support-tone model: gemini-2.0-flash prompt: prompts/support_v3.txt cases: - input: "Where is my order #4182?" graders: - json_schema: schemas/order.json - max_latency_ms: 2000 - input: "I want a refund, I'm in Germany." expect: "mentions the 14-day EU return window" graders: - contains: "14 days" - judge: faithfulness - 02
Run it.
One job per case, so a 500-case suite that dies at case 400 resumes from 400. Every call is cached on a hash of prompt, input, model and params — the second run costs nothing.
terminal$ diffscore run suites/support-tone.yaml 60 cases · gemini-2.0-flash ████████████████████ 60/60 cache hits 38 model calls 22 cost $0.31 elapsed 14.2s ✓ run_d41b07 stored - 03
Read the diff.
Improved, regressed, unchanged, against whichever run you pick as the baseline. With a p-value on the difference and an exit code your CI can act on.
terminal$ diffscore diff --baseline run_8f2a1c REGRESSION 4 cases p = 0.02 − refund-window-eu 1.000 → 0.000 − refund-policy-us 1.000 → 0.250 − escalation-tier2 1.000 → 0.000 + order-status-json 0.000 → 1.000 exit 1
Paired significance testing
A count is not a verdict.
Two cases flipping is noise. Four cases flipping on a 60-case suite might not be. Every other tool in this category shows you the raw count and lets you guess which one you are looking at.
Both runs execute the same cases, so the comparison is paired. That makes the right test McNemar’s on the pass→fail and fail→pass counts, or a bootstrap confidence interval on the mean score delta when the grader is not binary.
Diffscore reports the verdict first and the counts second, then hands your CI an exit code that matches. It is a small amount of code and a large amount of credibility.
In your pipeline
- name: Prompt regression tests
run: diffscore diff --baseline main
# exits 1 on a significant regressionWhat the terminal says
Sample dataSignificant regression on 4 cases
Four cases flipped pass→fail and none flipped back. On 60 paired cases that is unlikely to be chance.
exit 1CI failsNo significant change
Two cases flipped each way. The counts moved; the suite did not. Shipping this is fine.
exit 0CI passesThe part nobody else ships
We measure the judge, too.
An eval tool whose scores cannot be trusted is worse than no eval tool, because it manufactures false confidence. So every LLM judge in Diffscore carries its agreement with your own labels, printed beside it, including when that number is bad.
Judge reliability
judge:faithfulness- Verbosity correlation
- flagged above 0.300.11
- Variance over 3 runs
- 20-case sample± 0.04
- Self-preference
- model ≠ judgeblocked
Started at κ 0.54 when the rubric just rubric asked for “is this good?” Rewriting it into named criteria with required evidence is what moved it.
Sample dataDiffscore says so when the judge is unreliable. Below 0.6 the grader is marked untrusted and its scores are excluded from the headline verdict until the rubric is rewritten.
Every verdict cites evidence
The judge must return a score, the criterion that failed, and a quote from the output. A judge that cannot point at the text is usually hallucinating its verdict.
{
"score": 0.0,
"failed_criterion": "faithfulness-to-context",
"evidence": "You can return it if you change your mind. Let me know if you need anything else."
}Retrieved context said: “EU customers may return any item within 14 days of delivery for a full refund.”
Four known biases, four controls
In a pairwise comparison, whichever output comes first wins more.
Run both orders. Keep only the verdicts that agree.
Longer answers score higher regardless of quality.
Report score-vs-length correlation per grader. Flag above 0.3.
A model rates its own output generously.
Refuse to judge a model with itself.
The judge is told which prompt version produced the output.
Blind it. The judge sees input and output, nothing else.
Temperature zero is not determinism. Diffscore judges a sample of cases three times and reports the spread, so you can see the noise floor instead of assuming there isn’t one.
Grader escalation
Reach for the cheapest grader that works.
An LLM judge should be the last tool you pick up, not the first. A suite where most assertions are deterministic runs in seconds, costs nothing and never drifts — so that is the default Diffscore nudges you toward.
- 01
Deterministic
Free · instant · never driftsExact and fuzzy match, regex, JSON schema validity, required and forbidden strings, a valid tool call with the right arguments, latency and cost ceilings.
equalscontainsnot_containsregexjson_schemamax_latency_msmax_cost
Target share
70% - 02
Reference-based
Cheap · no judgement involvedEmbedding similarity or ROUGE against a known-good answer. Use it when the shape of a correct answer is known but its wording is not.
embedding_similarityrouge_l
Target share
20% - 03
LLM judge
Last resort · must be calibratedOnly for what the first two genuinely cannot express: tone, faithfulness to retrieved context, helpfulness. Every judge ships with a kappa beside it.
judge:faithfulnessjudge:helpfulnessjudge:tone
Target share
10%
Graders are plugins. The interface is one function returning a score, a verdict and a reason, so adding your own is a single file — and contributing it back is a pull request.
Install
Three lines to a verdict.
Bring your own provider key. Diffscore encrypts it at rest, never logs it, and never returns it from an endpoint.
- 01
pip install diffscorePython 3.12 or newer.
- 02
export GEMINI_API_KEY=...Or OpenAI, Anthropic, Groq, OpenRouter, or a local Ollama.
- 03
diffscore run examples/support.yamlShips with a worked example, so there is something to run immediately.
Then put it in CI
name: prompts
on: [pull_request]
jobs:
diffscore:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install diffscore
- run: diffscore run suites/support-tone.yaml
- run: diffscore diff --baseline main
# non-zero exit on a significant regression
env:
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}The exit code is the whole integration. No app to open, no webhook to configure — a regression fails the pull request the same way a failing unit test does.
Scope
What it deliberately isn't.
Langfuse, Braintrust and Promptfoo all exist and all do more. Diffscore does one thing — pre-deploy regression tests with scoring you can verify — and everything below is a thing it will not grow into.
No prompt playground
Every LLM tool has one, it is where the time goes, and it is not what makes a change safe to ship.
No production tracing
That is the observability category. Diffscore runs before you deploy, not after.
No team accounts in v1
One developer, one API key. Roles and billing are not what is failing on your team right now.
No hand-rolled providers
LiteLLM gives breadth for free. Four providers are tested properly rather than twelve claimed.