A new Slop University paper audits a martial-arts franchise's belt-grading checklist against independent guest examiners across twenty-four dojos, finding the checklist consistently clears more students than examiners do, and that a specific kind of disagreement predicts who stops training.
A season-long audit inside one franchise, and what happens to the students its checklist and its guest examiners disagree about.
A franchise-wide checklist is meant to make one dojo’s black belt mean the same as another’s. A new Slop University paper finds it mostly does — except in the cases where a guest examiner overrules it, which turn out to be exactly when a graded student tends to disappear.
The finding sits inside the School of Continuous Improvement’s standing interest in what an internal instrument quietly comes to stand in for once nobody checks it against the judgement it was actually meant to support.
An instrument that outperforms the judge it was built to agree with isn’t malfunctioning. It’s telling us what the checklist was actually for, which was never quite agreement.
— Dr Dagny Okafor, Lecturer and Convenor, Evaluation of Evaluation
Dr Torun Ezeigwe said the season’s data “raises a question we can’t settle from the inside — whether a different threshold, or a differently trained examiner corps, would close the gap without losing whatever standardisation the checklist is buying the franchise in the first place.”
The pair worked with twenty-four dojos across one franchise’s colour-belt programme over a forty-week season, comparing the outcome implied by each site’s headquarters checklist against the verdict of a guest examiner rotated in from outside, then followed graded students for twelve weeks afterwards to see who was still training.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.6k9y5a.