The School of Continuous Improvement has turned its own review apparatus into a subject of study, following two dozen staff through three stages of its internal practice-review cycle to see how what a panel notices tracks against what actually changes underneath.
The team spoke with participants shortly before the cycle opened, partway through, and again once outcomes were confirmed, coding each conversation for two things: language that mirrors the review’s own published rubric, and self-reported change to the practice the review is meant to assess. The first climbed sharply across the cycle’s middle stretch; the second barely moved, and held to the same pattern whether the conversation happened over video or in person. The School situates the result inside its longer-running effort to trace how a measurement instrument settles into the behaviour of the people it measures.
“The review was built to tell us something about practice,” said Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab. “What it’s told us this time is mostly about the review.”
“None of what we heard was staff acting in bad faith,” said Dr Anneke Tolan, Senior Research Fellow at the School of Emergent Priorities. “It’s what any of us would learn to do, given enough cycles — and it leaves the School with a fair question about what the panel should be looking for instead.”
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.62tny0.
Read the output: Reading the Panel, Not the Practice: A Mid-Cycle Interview Study of Signal-Learning During the School of Continuous Improvement's Own Review Cycle (doi:10.5555/slop.62tny0)