A new paper from the School of Continuous Improvement surveys 47 show-society produce judges across one season, finding a sealed pre-scoring hunch predicts the finished scoresheet closely, and that requiring every rubric column doesn't change how often the two agree.
A 19-society survey ties a judge's sealed walk-in hunch to the finished scoresheet, and finds full rubric compliance changes nothing
Slop University has published a new paper testing whether a show-society scoresheet decides a produce class’s placing, or simply writes down one a judge has already reached. The paper comes from the School of Continuous Improvement, whose Adaptive Metrics Lab and Evaluation of Evaluation program have spent recent cycles asking the same question of the University’s own review panels.
Associate Professor Casimir Beng, Dr Dagny Okafor, and Dr Thandiwe Solberg surveyed 47 judges across 19 regional show societies over one show season. Before opening a class’s scoresheet, each judge sealed their expected placing and a confidence rating from the first walk along the bench; the University then compared that sealed hunch against the completed rubric for 312 judged classes. First place matched exactly in 88% of them, and a judge’s stated confidence tracked the eventual scoresheet margin closely.
A rubric filled in top to bottom can still just be recording what a judge already knew from the first look along the bench.
— Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab
“We keep building instruments to check whether our other instruments are actually deciding anything,” said Dr Dagny Okafor, Lecturer and Convenor of the Evaluation of Evaluation program. “It’s a small comfort to find one, for once, outside our own buildings.”
The team also tested whether requiring every rubric column, rather than a quick total, changed how closely a judge’s hunch and final placing lined up. It did not: judges who completed the full scoresheet agreed with their own walk-in call about as often as those who moved straight to a number.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.a7p1c3.