A new paper from the School of Emergent Priorities compares two car-park boom gates that read the same automated confidence score, finding staff scrutiny drops sharply in the top band only at the gate where the score is shown to them.
A 13-week two-gate audit finds the display barely moves overall intervention, but empties the top confidence band of scrutiny
Two boom gates sit a few hundred metres apart on Slop University’s western campus, reading the same plates against the same automated confidence score. Only one shows a security officer that score on a screen. A term-long audit from the School of Emergent Priorities finds that difference alone moves how closely officers check the vehicles the system rates most confident — without moving, on an independent count, how often those vehicles turn out to actually belong there.
Researchers reconciled thirteen weeks of console activity across the two gates against a desk audit of a sample of admitted vehicles, carried out blind to what had happened at the boom, and against a staff-rotation check to rule out any one team’s habits. Overall intervention barely differed between the gates. Within the system’s highest confidence band, though, console-linked checks all but stopped at the gate where staff could see the score — while the same band’s vehicles were no more likely than any other’s to pass the independent audit.
A programme can keep reporting the same falling number long after that number has started meaning something different.
— Dr Marek Solheim, Lecturer, School of Emergent Priorities
Dr Anouk Mensah, the paper’s co-author, said the finding fits a pattern the School keeps meeting elsewhere in its own research: “the language of an automated system survives a lot of quiet re-scoping before anyone checks whether the substance kept up with it.”
The finding, the School notes, extends a running practice of auditing an instrument’s confidence against what it can actually confirm before anyone leans on it further. The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.j1l7f6.