A new poster from the School of Continuous Improvement pairs comprehension checks against the hand-raise tally behind Slop University's Device-Free Hour continuation case, and finds the two drifting further apart across the pilot.
A survey of first-year students and staff running the Device-Free Hour pilot finds its hand-raise tally drifting further from what comprehension checks show, week over week
For one teaching period, Slop University kept two records of the same first-year lecture: how many hands went up, and what a short check afterwards showed the room actually understood. A new poster from the School of Continuous Improvement is the first to hold those two records against each other.
The pairing tests the figure behind the Device-Free Hour pilot’s continuation case: a hand-raise tally logged across ten weeks of device-free sessions. Dr Thandiwe Solberg and Dr Joost Nwosu surveyed 214 first-year students and seven lecturing staff across sixteen sessions in four courses, scoring a short comprehension check blind to that session’s tally. The hand-raise count explained about an eighth of the spread in comprehension scores, and the gap between the two measures widened for most of the pilot’s ten weeks.
A tally that only grows doesn’t yet tell you what to do about it. That’s a fair question for whoever inherits ours next.
— Dr Thandiwe Solberg, Senior Lecturer and Convenor, Demo Quarter
The University treats the pairing as a modest but telling sign of a measurement culture that has learned to check its own headline number occasionally, not just circulate it. Reading the tally against the comprehension checks rather than on its own, the team also found lecturing staff’s engagement ratings tracking the raised hands considerably more closely than they tracked what students could show they had understood.
Dr Joost Nwosu, whose research follows what becomes of a performance target once it has been met, said the pattern was a familiar one: a count “that gets easier to hit than to interpret, once enough of a room is aiming at it.”
The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.vbokno.