A new Slop University paper puts a rural shire's noticeboard display standard through the test it has never faced, and finds the same board on the same day can be scored twenty-two percentage points apart depending on which officer is on the run.
Six officers classifying the same 420 notices agree barely more often than chance, and the compliance figure the shire publishes moves with them.
Ask six council officers to sort the same noticeboard and you get six compliance figures for it. Three Classes, Six Answers, published today by the School of Emergent Priorities, put a rural shire’s Public Noticeboard Display Standard to a test the Standard had not previously faced: whether two people applying it to the same notice arrive at the same answer.
Dr Anouk Mensah, Dr Iben Chikere and Professor Verity Marris photographed all thirty-four of the shire’s public noticeboards fortnightly for fourteen months, then asked the six customer service officers who perform the sweeps to classify a stratified sample of 420 notices independently, working from the Standard’s own definitions. Agreement across the six sat at a Fleiss’ kappa of 0.28. The scheme held together for shire notices and came apart on the two classes that describe what the district is actually doing, with fundraising items splitting almost evenly between them.
The classification does its work whether or not it repeats, and we thought that was worth establishing rather than assuming. A raffle is an activity of a local group and it is also goods offered for payment, and the board has never had to choose.
— Dr Anouk Mensah, Postdoctoral Fellow, School of Emergent Priorities
Recomputing each board’s published currency rate from each officer’s own classification produced figures spanning a median of twenty-two percentage points for the same board on the same day. “The School has learned something here about the difference between a figure that is reported and a figure that is reproducible,” said Professor Verity Marris, Director of the Trajectory Analytics Group.
The University regards the work as a further step in its anticipatory capability agenda, and as an indication of the measurement questions the School continues to open in ordinary settings. The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.m1cm77.