Ratchet

Improving an LLM prompt usually looks like guessing: edit the wording, run a few examples, eyeball the output, convince yourself it's better. There's no floor, so a change that quietly breaks one case while fixing another can slip through; and there's no bar, so a "win" might just be the new prompt leaking the answer or memorizing your test set. I wanted prompt work to behave like the rest of engineering, where a change has to clear a measured bar against ground truth before it counts. Ratchet is that gate.

Ratchet is a small scaffold for evaluating and improving prompts against your own labeled data, built around the guarantees that make the numbers trustworthy. A stable-hash train/holdout split keeps the same examples on the same side of the line across runs, so results stay comparable. Overfit and anomaly guards watch for a candidate that wins on the training slice but not in general. A regime fingerprint refuses to pool results across changed conditions, so swapping the model or the dataset can't silently invalidate a comparison. And a hill-climb whose winner must survive a held-out gate means nothing is trusted until it generalizes to data it never saw.

Each project wires in through five small seam adapters: a config, ingest, a runner, mutations, and the base prompt. The discipline cuts both ways: the same rig that certifies a real win is the one that exposes a fake one.

The loop's shape matches Andrej Karpathy's autoresearch, though I arrived at it independently: an LLM proposes a change, you score it against a frozen eval, keep the winner only if a number improves, and let it run overnight. Karpathy points that at ML research, where validation loss is objective and regenerable, so it needs almost no guards. Ratchet aims the same skeleton at ground truth that can lie, human grades and telemetry labels you can't re-mint. Same loop, opposite trust model; that's why the guards are the product.

Ratchet grew out of Rubrica, my AI exam-grading tool. I needed to know whether a change to the grading prompt actually moved grades closer to a teacher's, so I built the loop there first: ground truth is the released grade, the baseline is the AI's first-pass grade, a candidate is a grading-instruction addendum. On one problem the loop found a generalizing leniency addendum (training error 0.50 to 0.30, holdout-gated against the real grader); a free read-only probe then swept all six problems, found the grader already matched human grades, and saved a paid run by proving most needed defending, not improving.

The first time Ratchet left Rubrica was a completely different problem, the real test that it generalizes: a VLM reading chest-cam mountain-bike footage, grounded with GPS telemetry. Grounding had already lifted direction accuracy from 64% to 95.6%, but a stubborn residual remained: 12 of 134 climbs were still read as downhill even though the prompt stated a +11% gradient. The model had the answer in front of it and overrode it with what its eyes saw. That's an instruction problem, not a data problem, and instruction problems are what Ratchet is for. Ground-truth labels came for free; the synchronized GPS track already carries each frame's true direction in its gradient sign.

I hill-climbed on a balanced 100-frame dev subset for speed, testing three structural fixes (state the gradient sign first; an explicit sign-to-direction mapping; a camera-tilt resolution note), each adding structure rather than a louder nag. The winner was a single prepended instruction: "First read the gradient value provided above and state its sign, then give a direction that matches that sign." All three mutations independently reached the same score, which is the real lesson: the fix is to make the model follow the gradient, not any particular phrasing. The escalation gate passed cleanly (holdout gap of -0.019, no overfit or anomaly flags); the improvement generalized instead of memorizing the dev set.

The dev win held at full scale and shipped into the shared production preamble: 97.7% accuracy, uphill recall a clean 100%, climbs-read-as-downhill at 0. The cost: flat recall dipped from 85% to 82%, since committing to a sign pushes a few near-zero-gradient frames off "flat." Net was clearly positive, and the gate is what surfaced the trade at all; it measures the whole error, not only the part that improved.

The two accuracy points aren't the headline. The headline is that prompt quality became a ratchet: every change scored against ground truth and gated on held-out data before it shipped. The habit changes from "the new prompt feels better" to "it beat the old one on data neither was tuned to, and here's the number." The limits stay on the page too: validated on a single model and backend, coarse uphill labels, and the search ran on a dev subset while the reported figure is full-bench.