A new Slop University paper classifies the shortcuts a swim school's badge-test video reviewers take as a term-end backlog builds, finding badge outcomes hold steady while per-criterion feedback quietly disappears.
A term-long audit of 1,842 badge-test video reviews finds pass rates untouched by four different reviewing shortcuts, while specific feedback on the rubric quietly thins out as the backlog builds.
A four-minute badge review and a forty-minute one can still end in the same pass. A new Slop University paper traces where the other thirty-six minutes go.
Dr Ronja Oyelaran and Dr Marek Solheim, of the School of Emergent Priorities, sorted 1,842 end-of-term badge-test video reviews from a suburban swim school into four categories drawn from the review platform’s own session telemetry: a full rewatch, a single reference-clip check, a fast scrub through the footage, and a comment reused from an earlier review. Badge outcomes were statistically indistinguishable across all four, even after swapping the platform’s logged review time for durations independently re-timed from stored screen recordings. The feedback moved instead: the share of rubric criteria drawing a specific, non-boilerplate comment fell from 91% in the term’s first week to 58% in its last, tracking that week’s submission volume closely.
Passing is the part a family checks. We would like to know whether they’d notice the comment field thinning before they noticed anything about the badge itself, and if a backlog is going to cost a swimmer something, we’d rather it wasn’t decided by whichever four minutes of tape a reviewer had left.
— Dr Ronja Oyelaran, Senior Lecturer, School of Emergent Priorities
Dr Marek Solheim, the paper’s co-author, said the open question was the one the team wanted to chase next: “whether five reviewers, or eight, would have kept week eight looking like week one is the trial we’d actually want to run.”
The comparison design borrows from two earlier University audits — one tracing a guarantee’s shifting clock-start, another checking a grading checklist against an examiner’s verdict — with a review panel’s own workload standing in as this run’s variable.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.969bce.