A new Slop University paper calibrates the 24-item checklist behind an accredited first-aid practical assessment and finds it is not the same instrument at the six training providers whose first-attempt competency rates are published for comparison.
A Rasch analysis of 9,148 assessments finds 11 of 24 checklist items behaving differently across the six providers a safety scheme lists side by side
Six first-aid training providers sit side by side in a public directory, separated by three and a half percentage points. A Slop University paper released today reports that the tick sheet those percentages are calculated from does not behave the same way at any two of them.
Everyone Passes, Differently, released through the Office of Research Outputs, takes the 24-item observation checklist an assessor completes beside a resuscitation manikin and calibrates it as a measurement instrument, across 9,148 completed assessments gathered over nineteen months. Nineteen of the 24 items are ticked on more than 97 per cent of sheets. Eleven behave differently depending on which provider ran the course.
This School has spent a great deal of its effort asking whether a number is right. What this one has taught us about ourselves is that a number can be right in six places at once and still not let you put the six in order.
— Dr Torun Ezeigwe, Senior Lecturer and Director, Master of Applied Measurement
The team paired the providers’ own assessment records with a pre-registered substudy in which a second marker, working blind, observed 612 of the same assessments independently. Rebuilt from those observations, the checklist separates candidates roughly four times as well as the administered version does. Re-anchoring the scale on the 13 items that behave consistently everywhere moves four of the six providers by at least two places, with the provider carrying the highest published rate settling in fourth.
“The question we would like somebody else to take up is what an assessor’s sheet would have to record for this to be answerable without a second person standing in the room,” said Dr Solveig Adeyemi, Lecturer in the School of Continuous Improvement.
Slop University holds that a figure offered up for comparison carries an obligation that a figure merely reported does not, and regards the testing of that obligation as ordinary work rather than a special occasion. The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.czhkle.