Four blind assessors rated the same 312 end-of-term learn-to-swim assessments for a new Slop University research poster, which finds the operator's six-stage framework separating reliably into three bands, with most recorded progression falling inside the assessors' own disagreement.
Blind re-rating of 312 end-of-term assessments across nine centres puts assessor agreement at fair over six stages and substantial over three
Four assessors, none of whom had taught the swimmer, watched the same recorded end-of-term assessment and wrote down a stage. Slop University has published what they wrote. Six Labels, Three Bands, a research poster from the School of Emergent Priorities, reports the agreement between 1,248 such ratings gathered across nine centres of a metropolitan leisure-centre operator.
The University regards the study as a considered contribution to the measurement questions the School has made its own, and is glad to see it deposited.
Six names on a chart is a promise that six things can be told apart. Our assessors could tell three apart, reliably and in good faith, and we would welcome an argument that we have counted the bands wrong.
— Dr Lindiwe Achterberg, Lecturer, School of Emergent Priorities
The study rests on deck-recorded assessments gathered over two teaching terms and re-rated blind by teachers rostered elsewhere in the network, with the band structure and its boundaries settled before any rating began. Agreement across the six published stages is reported as fair; under the three-band reading it becomes substantial, clearing 85 per cent at every stage.
Dr Ronja Oyelaran, Senior Lecturer in the School, said the finding “sits with the framework rather than with the people applying it”, and noted that a majority of the movement the operator’s progression rate records falls between stages the assessors did not reliably separate.
The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.duve1d.