Rubrica
Privacy-first AI exam grading. Keeps stored names, SIDs, and cover pages out of external grading requests, grades via Claude's vision API against decomposed rubrics, and surfaces class-level learning gaps.
Hand-grading doesn't scale; the obvious AI fix outsources FERPA to a vendor
As a Graduate Student Instructor at Haas, I was grading 100+ handwritten exams a term: slow, inconsistent across a stack, and impossible to standardize by hand. The obvious fix, an off-the-shelf AI grader, means uploading identifiable student work to a third-party tool, where FERPA compliance rests on the vendor's contract rather than on the architecture. So the binding constraint was privacy before accuracy: student identities had to stay under the instructor's own control and out of every AI model's reach.
Decompose the rubric instead of reaching for a bigger model. Rubrica splits each rubric into structured per-question partial-credit tiers. In the one human-reference comparison so far, the decomposed section disagreed with human grades far less than the single holistic essay (1.26 vs 3.88 points per exam), consistent with that approach but not yet causal evidence. A related discipline fell out of it: models reliably start dropping instructions past roughly 30 directives, so prompts are split into multi-step pipelines rather than one giant prompt.
Put the privacy boundary in the architecture. Name-mapping happens locally. Under the default local-Ollama configuration, stored names, SIDs, and cover pages are excluded from external grading requests, which carry only an anonymous ID, answer-page images, and rubric text; answer pages and rubrics can still contain incidental identifiers, so the instructor reviews for those before grading. The only thing that carries a name off the machine is a report the instructor sends from their own Gmail. And whatever checks a grade is a different model from the one that produced it: a primary Claude grader, a reference model from another vendor that re-grades a stratified sample as a structural check, and escalation to Claude Opus 4.7 on ambiguous multiple-choice reads. That reduces one source of correlated error; it is not third-party independence.
Rubrica reads scanned exams, anonymizes each one with a random ID, and grades against a rubric that has been decomposed into structured per-question partial-credit tiers with common error patterns and expected methods. Each criterion is scored independently by Claude Sonnet 4.6's vision API, then mapped back to the original student locally. Stored names, SIDs, and cover pages never enter the grading request; answer pages and rubrics can still carry incidental identifiers, which the instructor reviews for before grading.
After grading, the system generates class performance diagnostics that surface which concepts the cohort missed and where partial credit clustered, and the comparison's review checklist flags which rubric items drew the largest cross-model disagreement. That's what I use to plan review sessions and tighten the rubric before the next exam.
Rubrica has been used across microeconomics, finance, and data & decisions courses at UC Berkeley Haas; the validation evidence here covers one economics course and one macroeconomics midterm. The most recent reported cross-model comparison (run March 2026), author-run and matched-input, covered 30 exams (840 scored items) from an undergraduate economics course. Each exam was re-graded by o4-mini as a reference scorer from another vendor, which reduces, without removing, the shared-training and shared-architecture confounds; the models received matched rubric content and rendered answer pages, with provider-specific request formatting, so this is a matched-input comparison rather than an experiment in which the model is the only variable. The in-app check now defaults to Gemini 3.1 Pro, still a different vendor from the grader. The production model reached a consistency ICC of 0.966 (95% CI 0.93 to 0.98; earlier versions reported 0.964 from a calculation error), a descriptive range rather than a certification threshold, and QWK of 0.883, above the 0.70 e-rater benchmark, which is task-dependent. Mean absolute error: 0.15 pts per question. The observed production-minus-reference mean difference was +0.64 points per exam, with an approximate 95% interval (−0.49 to +1.76) that spans zero, so its direction is uncertain. All nine letter-grade mismatches in the comparison were between adjacent letters, and seven sat inside the window a boundary re-grading safeguard targets: it triggers a second grading pass on any exam scoring within ±1.5% of the 90/80/70/60 letter-grade cutoffs. The other two fell outside it.
That comparison measures model-against-model concordance, a structural check, not accuracy against a human grader. The first comparison against human-assigned reference scores landed in July 2026: a macroeconomics midterm from another instructor, with human scores recorded in Gradescope, 25 exams joined per question. On the decomposed questions (46.5 points across 38 items) Rubrica sat within 1.26 points of the human score on average, 2.7% of the section, an observed mean difference of −0.28 points, with 85.4% of compared cells (810 of 949) matching exactly. The one holistically scored item, a 25-point essay, had a mean absolute difference of 3.88 points. On this sample the decomposed section disagreed less than the essay, yet item format, point scale, and content all differ, so the comparison does not isolate the cause. QWK and ICC are calculable at n = 25; their precision is limited, so a larger rerun is planned.
A third study bounds run-to-run noise. In June 2026, ten stratified final-exam submissions were each graded three times through the production pipeline: all ten kept the same letter grade, with a mean within-exam total-score standard deviation of 0.91 points. One workspace and three repeats, so it measures test-retest stability, not accuracy.
Beyond boundary re-grading, the pipeline runs MC double-read verification on zero-scored multiple-choice items, contradiction detection between feedback and scores, feedback specificity enforcement on vague comments, handwriting confidence flagging for ambiguous pages, and a review-flag gate that prevents finalization until an instructor dismisses each unresolved flag. A pre-safeguard raw-score snapshot is preserved so cross-model comparisons can use first-pass scores.
Rubrica generates a per-student PDF report (score card, question breakdown, rubric-anchored feedback) and emails it from the instructor's own Gmail through a one-click Connect Gmail button, with no Google Cloud Console or developer account to set up. Mail sends locally from the instructor's account, so student PII never routes through a server; delivery is tracked per recipient, and there's a disconnect-and-revoke path.
Rubrica now ships as a standalone macOS app: a Tauri shell wrapping the Flask backend, with the React interface, templates, and SQLite frozen into a single bundle. A non-technical instructor installs a .dmg and runs it like any other Mac app, with no terminal, virtualenv, or hand-edited config file. On first launch the instructor enters an Anthropic API key, then an optional setup screen walks through the on-device OCR model, the class roster, and report email; on macOS 14 or later the local engine that reads cover pages ships inside the app, so there is nothing extra to install. The current pilot build, 0.1.10 (October 2026), is Developer ID signed and notarized, runs on Apple Silicon Macs with macOS 12 or later, and checks a version manifest for updates from Settings; the first pilot build went to outside testers in late August 2026.
Because grades and rosters are sensitive, the app gates access behind a one-click Sign in with Google (OpenID Connect); only allowlisted identities can open a workspace. The first sign-in claims ownership, and the owner adds or removes GSIs by email from Settings, so a teaching team shares one instance without sharing credentials. ID tokens are verified (audience, issuer, expiry, verified email) before any session starts. Distributed builds add two more layers, enforced since the August 2026 release: after sign-in the app exchanges the Google token for a short-lived signed entitlement from a server I control (allowlist by email, a seven-day lease honored offline), and an operator PIN is required every time the app opens, with a nav Lock button to lock it on demand. A per-launch boot id makes sure no cached session skips the PIN; Google sign-in itself recurs only when the lease lapses. The license check carries no student data, so it changes nothing about where exams live.
This page is the summary. The assistant in the corner has the detail behind it, so ask how the validation studies were run, what the rubric-granularity finding means, or where the pilot stands.
The 0.1.10 pilot build has a versioned Zenodo record, typed as a report rather than software, so Rubrica can be cited without pointing at a page that will keep changing: the software description and a validation note that states what each of the three studies does and does not measure (the studies ran on development builds between March and July 2026 and were not re-run). This version also states what 0.1.10 changed in grading: how a question returned more than once and a total with no per-question scores are handled, and how an instructor's score edit settles a borderline average. The grading prompt and scoring rules for an ordinary response are unchanged. An earlier version of the record corrected an ICC calculation error (0.964 is now 0.966). The DOI in the entry is the concept DOI, which always resolves to the current version of the record. The full March 2026 comparison report is available on request.
Fraser, T. (2026). Rubrica v0.1.10: Software description and validation report [Report]. Zenodo. https://doi.org/10.5281/zenodo.22313469