The 1-hour eval harness that saves you on model upgrades

If you run an AI product in production, the most insidious failure moment isn't when you write code — it's when your provider ships a new model. The new model is better on almost every metric. But it silently breaks a slice of the prompts, few-shot examples and edge-case patches you spent months tuning to the old one. Nothing throws. You just hit a weird downstream symptom a few days later.

The reflex fix is 'let me write a better prompt.' Wrong layer. What you actually need is a small eval harness that tells you in 60 seconds whether the model still behaves. It takes an hour to set up, not a weekend. Here's the production-tested version.

1 — Build the test set from your own logs.

Don't generate synthetic test data. Pull 20-50 real cases from production logs. Not the ordinary ones — specifically the weird outputs, the cases where users fed unexpected input, the ones the model previously screwed up. This set is your product's real distribution; synthetic data never captures it.

• ▸20-50 cases is enough; not thousands. The goal is signal, not coverage.
• ▸Always include cases that broke before or generated complaints.
• ▸Store each case as (input, expected invariant) — not the expected exact output.

2 — Assert invariants, not exact strings.

The biggest beginner mistake: expecting the output to equal a string verbatim. The model won't always reply word-for-word the same, and it shouldn't have to. You want to test behavioral correctness, not stylistic sameness. Assert the invariants a correct output cannot violate.

// Bad: brittle, breaks the moment phrasing shifts
expect(out).toBe("Done, your appointment is booked for 2pm.");

// Good: tests behavioral correctness
expect(out.toolCalls).toContainEqual({ name: "book_slot" });
expect(out.charges).toHaveLength(1);        // no double charge
expect(() => JSON.parse(out.payload)).not.toThrow();
expect(out.amount).toBeGreaterThan(0);

Typical invariants: was the right tool called? Was the customer charged twice? Is the output valid JSON? Is the amount positive? These are style-agnostic and catch the real failures.

3 — Make the failure output readable.

When an assertion fails, 'expected true, got false' is useless. The failure message alone should let you diagnose: which input, what the model did, which invariant broke, what the raw tool args were. A good failure line explains the problem without you ever opening the logs.

// A failure line should read like a log line:
// CASE #37 | input: {user_id: null, amount: "12.5"}
//   → expected 1 tool call, got 0
//   raw args: {"action":"clarify","reason":"missing user"}
//   broken invariant: must call book_slot when amount present

4 — Run both models, take the DIFF.

This is where the real leverage is. Run the same set on both the old and new model and diff the results. You'll find the new model matches on 46 of 50 cases and regresses on 4. Those 4 cases are exactly your re-tuning list. Instead of blindly 'a new model dropped, let me review all my prompts,' you know the precise spot to touch.

“The upgrade decision is now a table, not a gut call: 46/50 stable, 4 regressions — and you know each of those 4 by name.”

5 — Every prod incident becomes a permanent test.

When a new bug shows up in production, add that case to the harness as a test BEFORE you fix it. Over time the harness becomes the memory of every real failure your product has ever had. That's a moat nobody can copy: a competitor can steal your prompt, but not your six-month-deep library of incidents.

The beauty of all this: it's an hour to set up, but it saves you hours of debugging on every model upgrade and every prompt change. Hardening a prompt is one-off; the harness is permanent.

“The model was never the weak link. The weak link was not having a layer that verifies it.”