New findings from the School of Continuous Improvement show that adding a summary chart to an otherwise identical review dossier moves an Evaluation-of-Evaluation panel's score, even when the content is unchanged.
A matched-pair test handed panels the same dossier twice, once with a summary chart and once without, and watched the scores pull apart
Give an Evaluation-of-Evaluation panel two copies of the same dossier, identical but for a one-page summary chart, and it will score them as though they described different cases. That is the finding of a new study from the School of Continuous Improvement, which set out to test whether a review built to weigh substance can tell substance apart from the way it is presented.
Across three review cycles and fifty-eight panellists, the team paired each dossier with a chart-bearing twin and held everything else fixed, pre-registering both the chart assignment and the review’s own tolerance for score variation. The chart-bearing copy ran consistently ahead, by a margin the certified tolerance band reads as a real difference in quality, and reminding panellists partway through that the content had not changed did nothing to close the gap. In their written reasons, panellists in the chart arm reached for a figure that appeared only in the chart.
The School of Continuous Improvement frames the study as an early attempt to measure not what a review assesses but what quietly moves it.
A review that certifies how carefully a case was weighed doesn’t usually ask what did the weighing. We wanted to know whether a chart could stand in for the case without anyone noticing it had.
— Dr Osei Vandermeer, Research Fellow and Convenor of the Review Cadence Observatory
“The honest version of this is that we can’t yet tell a panel to ignore a chart and trust that it will,” said Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab. “Knowing the effect is there is the part we can act on.”
The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.gw64x7.