A Slop University paper treats a retail chain's mid-cycle performance-review scale migration as a natural experiment and reports the University moving to recommend four-point, no-midpoint scales wherever a workplace rating feeds an eligibility decision.
A retail chain's mid-cycle switch from a five-point performance-review scale to a four-point one gave the Evaluation of Evaluation program a ready-made test of what the redesign itself does to the people it rates.
A rating scale can change overnight, the way an app updates while nobody is looking. A paper published today by researchers from the School of Emergent Priorities and the School of Continuous Improvement’s Evaluation of Evaluation program treats one such change as a ready-made experiment: when a national retail chain collapsed its five-point review scale into a four-point one, dropping the option to call a rating “middle”, employees reviewed just after the switch lodged formal appeals at close to three times the rate of employees reviewed just before.
A scale is supposed to be the least interesting part of a review. This one turned out to be the only part that mattered.
— Dr Dagny Okafor, Lecturer and Convenor, Evaluation of Evaluation, School of Continuous Improvement
The paper compares reviews completed either side of the chain’s migration date, treating the changeover itself, rather than any employee’s underlying performance, as the dividing line between two otherwise similar groups, one of a small family of instruments the University uses for reading a routine software rollout as a controlled comparison.
Associate Professor Casimir Beng, whose Adaptive Metrics Lab traces what a measurement system does to the thing it measures, called the jump “about as clean a version of that story as we’ve ever been handed by an outside vendor’s software update.” On the strength of the finding, the Evaluation of Evaluation program is moving to recommend four-point, no-midpoint scales as the University’s preferred instrument wherever a workplace rating feeds an eligibility decision.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.j51e5k.