A new Slop University paper enumerates 2,304 defensible ways of computing a children's martial-arts association's term progression rate from a single grading register, finding estimates from 38.6 to 94.1 per cent against a published figure of 91.4.
Eight undocumented decisions put the same martial-arts grading register anywhere between 38.6 and 94.1 per cent
There are 2,304 defensible ways to work out how many children advanced a belt last term, and the association that publishes the figure has never said which one it uses.
School of Continuous Improvement researchers obtained a metropolitan children’s martial-arts association’s complete grading register for 18 school terms across 34 clubs, covering 8,911 students and 37,204 grading attempts, and computed its published progression rate under every combination of the eight decisions its four-sentence methodology note leaves open. The estimates run from 38.6 to 94.1 per cent. The association’s published 91.4 per cent is among them, returned by 74 specifications, all of which count only the students who turned up and knelt down to be graded.
We came to this expecting to check somebody else’s arithmetic and spent most of the year checking our own. The School has published rates with no more of a method note than this one, and that is the part we have taken away.
— Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab
Which denominator a specification adopts accounts for most of the spread; how repeat attempts are selected accounts for much of the rest, in a setting where a second attempt within the term costs $48 at most clubs and nothing at the others. Dr Renke Sabel, Senior Lecturer and Convenor of the Indicator Commons, said the register itself was in unusually good order. “Nothing here was recorded badly,” he said. “It was recorded for one purpose and then asked a different question, which is the ordinary life of a number.”
The University takes the paper as a sign of its schools working where the record is kept rather than where the claim is made.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.ed543f.