A School of Continuous Improvement paper scores 156 performance-review calibration sessions on a five-level maturity index and finds the score does not predict independently checked rating accuracy, though a partner firm now sets bonus-pool multipliers by it alone.
A five-level Calibration Maturity Index scored 156 review sessions at a professional-services firm, and now sets each unit's bonus-pool multiplier alone
A five-level instrument for scoring how a performance-review calibration meeting is run has been adopted by a partner organisation to set its own bonus-pool multipliers, the School of Continuous Improvement reports today. The Calibration Maturity Index, developed and validated across three annual review cycles at a professional-services firm, orders a session’s practice from unstructured discussion through to independently re-scored, blind-reconciled ratings. The School frames the result as part of its ongoing effort to see clearly what an instrument measures before an organisation leans its weight on it.
A calibration meeting can look identical to itself year after year and still tell you nothing new about the ratings underneath it, which is exactly the kind of quiet the School exists to notice.
— Dr Dagny Okafor, Lecturer and Convenor of Evaluation of Evaluation
The index was checked against a source it was never designed to answer to: an external panel’s blind re-scoring of the underlying employee evidence, independent of both the original ratings and each unit’s coded level. The resulting comparison is reported in full alongside the instrument itself, rather than held back for a later paper.
“The index behaved exactly as an index should,” said Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab. “What an organisation then does with that behaviour is a separate question, and one the Lab is glad to see asked out loud.”
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.oi3m67.