A new paper from the School of Emergent Priorities and the School of Continuous Improvement validates a book-club host-scoring instrument against an anonymous show of hands, finding good internal consistency but only modest agreement with confessed skips, nearly all of it carried by a single item.
A field validation across 214 meetings finds the five-item score's internal agreement outruns its accuracy, with nearly all its signal riding on one item
A five-item scoresheet can make a book club host’s private impression look like data. Whether it behaves like data — reliable, and also right — is the question a new paper from the School of Emergent Priorities and the School of Continuous Improvement puts to 214 real meetings across 27 neighbourhood book clubs.
The instrument, a five-item Skip-Likelihood Index a host fills in as discussion happens, agrees with itself well: its own items rise and fall together across the study’s eight months. Set against an anonymous show of hands taken at the close of each meeting, though, the index’s predictions land only loosely, and the paper traces almost the entire relationship back to a single item.
An index that agrees with itself is doing something. It just isn’t necessarily doing the thing you built it for.
— Runa Adegoke, Senior Lecturer and Convenor, Strategic Drift Survey
Dr Adegoke and Dr Renke Sabel set the host-scored index against a same-night confession count, collected independently and only after scoring was sealed, across a regional network of library-affiliated and independent clubs. A second host quietly repeated the scoring at a subset of meetings, to see whether two people watching the same room would flag the same attendees.
“It isn’t really about book clubs,” said Dr Sabel. “It’s about any scoring job a person does quickly, in a room, without asking directly, which describes a great deal of what this School spends its time watching.”
The University counts the space between an instrument’s internal agreement and what it actually predicts among the more useful gaps its research keeps finding, precisely because the gap is so easy to miss from inside the instrument itself.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.3hjzqw.