A School of Continuous Improvement paper develops and validates a four-domain rubric for judging Book Week costumes, finding good agreement between trained judges and no association between a costume's score and what it cost to make.
Nine primary schools trial an instrument built to score costume effort without quietly scoring the receipt
The instruction printed on the underside of a thousand Book Week judging clipboards is always some version of the same three words: judge on effort. Until this week, no primary school had a documented way to check whether its judges actually could.
A paper published today by the School of Continuous Improvement reports the development and pilot of the Effort Parity Rubric, a four-part scoring instrument for costume parades, trialled across nine primary schools after a single shared calibration session for the volunteer judges who applied it. Three independent judges scored every costume entered; their scores agreed well, and agreed just as well once the rubric’s one deliberately effort-signalling item, a family’s self-declared Home-Assembly Attestation, was quietly removed from the sum.
The interesting finding isn’t that judges can agree. It’s that they kept agreeing once we took away the one item that told them what answer we wanted.
— Dr Torun Ezeigwe, Senior Lecturer and Director, Master of Applied Measurement
Costume scores tracked neither what a family reported spending nor whether a costume was purchased outright, a result the design team reads as evidence the rubric measures something closer to craft and character than to a household’s discretionary budget. Dr Dagny Okafor, who co-led the instrument’s development, said the calibration protocol “matters more to us than any single point weighting on the scoresheet,” and that the weightings themselves are expected to shift as more schools’ parades contribute exemplars.
The University reads the pilot as an early but encouraging step in its wider programme of measurement-instrument development, and expects the calibration method, if not yet the rubric itself, to travel well beyond the nine schools that tested it.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.76v0zf.