A new Slop University paper validates a driving-school network's composite Practice Hour Authenticity Index against instructors' independent judgement, finding the oral-quiz component agrees far more closely than the motion-sensor tally it is blended with.
Validating a learner-driver logbook app's blended score against instructors' own judgement finds the oral quiz outperforms the motion sensor it's weighted against
A composite score is only as good as whichever ingredient it happens to trust. Ours, it turns out, has been trusting the wrong one.
— Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab, School of Continuous Improvement
A driving-school network’s Practice Hour Authenticity Index blends two things into one number: a phone’s own motion-sensor reading of a logged practice drive, and a short oral quiz asking the learner to describe one drive at random. A new Slop University paper puts the blend, and each half separately, against what the network’s own instructors already believed.
The gap is not close. Checked against 34 instructors’ independent judgement across 212 learners and 316 quizzed drives, the motion-sensor reading agreed with instructors only slightly (Cohen’s kappa = 0.11); the oral quiz agreed with them moderately well (kappa = 0.52). The index the network actually runs, weighted 70% toward the sensor, recovers to kappa = 0.18 — closer to its weaker half than its stronger one.
“The quiz was never the expensive part of this system,” said Dr Mirela Hanke, Postdoctoral Fellow, School of Continuous Improvement. “It was just the part somebody had to be on the phone for.”
The finding lands squarely inside the Adaptive Metrics Lab’s standing brief: instruments that measure what is cheap to collect rather than what actually agrees with the judgement they are meant to stand in for. An ablation stratifying by instructor years of experience left the pattern unmoved.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.8c99s8.