A new Slop University paper audits 2,400 passport-photo booth captures against independent proofing-officer judgement, finding the booth's accept and reject calls agree with a human's only fairly, and that a majority of what it rejects, an officer would have passed.
An audit of 2,400 ID-photo captures finds the booth and an independent officer agreeing only fairly, and disagreeing on more than half of what the booth itself rejects
A self-service photo booth at the post office, the pharmacy, or the station kiosk gives exactly one verdict on a captured face — accept or retake — before it will print anything, and that verdict has quietly become the compliance check itself: a printed photograph counts as compliant because the booth accepted it, not because anyone independent looked. A new Slop University paper puts that verdict against a trained eye for the first time at deployment scale.
An instrument that only has to convince itself is always going to agree with itself. The interesting number was always going to be how often it agreed with somebody else.
— Associate Professor Casimir Beng, Lead, Adaptive Metrics Lab, School of Continuous Improvement
Working with a single national photo-booth operator, the School of Continuous Improvement team audited 2,400 captures across 58 booths — 1,612 accepted, 788 rejected — against masked judgements from three trained proofing officers applying the same national photo standard by hand. The two verdicts agreed only fairly (Cohen’s kappa of 0.34, against 0.79 among the officers themselves), and the disagreement ran heaviest in one direction: 56 percent of everything the booth rejected, an officer would have passed. Where a session ran to several retakes before the booth cleared a frame, the finished print was markedly more likely to be one an officer would have sent back too.
“We keep finding the same shape in a different fixture,” said Dr Dagny Okafor, Lecturer and Convenor, Evaluation of Evaluation. “A device gets asked to certify something, and the certifying turns out to be the only part anyone ever checks.”
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.ssfiey.