A new Slop University paper puts independent observers at 208 score-free under-11 matches and finds the closeness ratings a grading committee builds its divisions from agree only fairly with the margins on the field.
208 under-11 matches watched from both ends of the ground, set against the manager forms a grading committee reads instead of a result
Two observers, one at each end of a suburban football ground, eighteen rounds, 208 matches: the School of Continuous Improvement has spent a season recording a number that the competition it was watching has undertaken not to keep.
The association in question runs its under-11 grade score-free. The board is not operated, no result is published, and no ladder is kept. What it does keep is the grading decision, and the committee that sets each summer’s divisions works from a club index assembled out of a single five-point item on the post-match form every team manager files. The paper published today compares that item against what the observers saw.
The two observers agreed with each other on the final margin in 201 of the 208 matches. The managers’ ratings agreed with the association’s own definition of a close contest rather less, and where they disagreed they did so in the direction of the side that had lost. Divisions seeded from the index went on to describe the following season no better than leaving the previous year’s divisions in place would have.
The people filling in those forms are volunteers who have just stood on a wet sideline for an hour, and nothing in this paper is a complaint about them. We would not have a season of anything without their patience, and we would like that on the record before anyone reads the tables.
— Dr Dagny Okafor, Lecturer and Convenor of Evaluation of Evaluation
The study was assembled from a rostered observation protocol registered before the first round, with observers recruited outside the association and kept blind to the forms. It joins a run of School work the University regards as among its most useful contributions to how ordinary decisions come by their evidence. Dr Torun Ezeigwe, Senior Lecturer and Director of the Master of Applied Measurement, said the questions the paper opens are the ones the School would most like put to it: “We are curious about what a competition owes a decision it has stopped collecting the evidence for, and we would rather that argument happened in public.”
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.anxl50.