A School of Continuous Improvement paper regrades a decade of capstones under every year's official rubric text and finds the assigned distinction band tracks the drafting committee's chair more than the work it graded.
Regrading a decade of Master of Applied Measurement capstones finds a graduate's distinction band tracks who chaired that year's rubric far more than the work itself
Every year, a rotating committee in Slop University’s School of Continuous Improvement redrafts the marking rubric for the Master of Applied Measurement’s capstone project, standard practice on a program built around the idea that what an institution measures, it can improve. A new paper asks what happens when a decade of capstones the rubric graded gets marked again, under every other year’s version of the same instrument.
Researchers regraded 140 capstones from ten MAM cohorts twice each, blind to the original grade: once under their own submission year’s rubric text, and once under a different year’s. When the alternate rubric shared the capstone’s own drafting chair, the distinction band changed for about one project in nine; when it came from a different chair’s tenure, it changed for more than half, a gap a separate, rubric-blind quality ranking barely narrowed.
The School of Continuous Improvement treats the finding as a discomfort worth sitting with rather than resolving quickly: the flagship program that trains the University’s own measurement specialists cannot, on this evidence, hold its central assessment instrument steady across the people who take turns drafting it.
We built this design to find out whether the rubric was reading the work or the work was reading the rubric committee. It turned out to be the second one, and that is not a comfortable thing to learn about your own flagship program.
— Dr Renke Sabel, Senior Lecturer in the School of Continuous Improvement
Dr Mirela Hanke, Postdoctoral Fellow in the School of Continuous Improvement, recognised the pattern from her own work on the University’s dashboards. “An indicator that migrates from measuring something to deciding something usually does it quietly enough that nobody notices the handover,” she said. “Here you can actually watch the distinction band cross over, cohort by cohort.”
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.jze8gt.