A study of 78 acquitted grants from one state regional-events fund finds nine different ways of filling in a single box marked attendance, and shows the fund's efficiency ranking follows the method rather than the crowd.
Forty-one funded community events counted twice, once by their own committees and once by a common protocol, and the fund's cost-per-attendee ranking reorders
Ask a funded festival how many people came and you will get a number. Ask nine of them how they arrived at it and you will get nine answers. Nine Ways to Count a Main Street, published today by the School of Continuous Improvement, takes one state regional-events fund’s acquittal record and asks what its central figure is a count of.
The team read the method statements attached to 78 acquitted grants, sorted them into nine classes, and then went to 41 of the events with a counting protocol of their own. The acquitted figure came in above the researchers’ estimate at 35 of the 41, by a median factor of 1.88. Which method a committee used explains 88.1 per cent of the difference; how big the event was explains 3.1 per cent.
Nobody in this study did anything wrong. The form asked for a number and every committee gave it one. What the form never said is what it wanted counted, and the nine reasonable readings of that word are now sitting in a ranking that treats them as the same.
— Dr Ingeborg Nwachukwu, Lecturer and Convenor, Capability Sprints
Recomputed on a common count, the spread in cost per attendee across methods narrows from more than threefold to a little over one. Fourteen of the 41 events change sides of the threshold the fund uses to decide who is invited back.
“A ranking is only as portable as the instrument underneath it, and this one was chosen locally and inherited annually,” said Dr Joost Nwosu, Postdoctoral Fellow. “We would rather that were known than corrected in a hurry.”
The University regards the study as a plain application of the measurement discipline its Schools were established to practise, and welcomes the attention it may draw to the boxes on other forms. The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.67ay7v.