A cluster-randomised trial across one council's grants round finds that groups given a hand tally counter and a distinct-person definition report a third fewer people engaged, and that the program's cost-per-participant ranking moves against them for it.
Sixty two funded groups randomised at award, and twelve of the thirty one issued a counter fell a renewal band on the figure it produced
Thirty-one community groups opened an envelope at the start of a funding round and found, beside the funding letter, a small metal counter and a single page setting out what counts as a person.
Counted Once, Funded Less, deposited today through the Office of Research Outputs, is a cluster-randomised trial run across one metropolitan council’s twelve-month community-grants round. Sixty-two funded groups were allocated at award: thirty-one received the counter and the definition, thirty-one received the acquittal form as it has always been. Both arms then reported, event by event, the one figure the council asks every grantee for.
The groups with the counter returned a median 152 people an event. The groups without returned 236. At twenty-four events an independent observer counted alongside; the counted returns came in six per cent above that observer, the uncounted sixty-eight.
We asked thirty-one groups to count more carefully, they did it, and the ranking then treated them worse for it. Whatever else this trial settles, the University owes those groups something better than a note about method.
— Dr Solveig Adeyemi, Lecturer, School of Continuous Improvement
The council divides each grant by the reported figure and ranks the round on cost per participant. Re-derived on the returns the trial produced, twelve of the thirty-one counted groups fell a renewal band, against two of the thirty-one that were never asked to count.
Dr Sten Okwuosa, Senior Research Fellow and Convenor of the Impact Pathway Atlas, said the School had been “interested for a while in the distance between doing a thing and being recorded as having done it”, and expected the trial to sit alongside its wider work on how recognition finds the work that earned it.
The University places the trial within a longer commitment to understanding what its methods do once they are handed to somebody else. The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.sxwqok.