A new Slop University research poster scores 384 recorded calls to a water utility's contact centre against the twenty-item rubric behind its service-quality figure, and finds near-total assessor agreement with no relationship to whether the caller's fault was resolved.
Three assessors scoring 384 recorded utility calls agree almost completely, while the score's association with resolution at fourteen days sits at 0.04
A caller reports a leak. Somewhere behind that call, three people who were not on it will listen to the recording and agree, closely, about how it went.
Agreement Without Resolution, deposited today through the Office of Research Outputs, asks what that agreement is worth. The School of Continuous Improvement scored 384 recorded calls to one water utility’s contact centre — leaks, bills, meter faults — against the twenty-item rubric the centre reports its service quality from, with three accredited assessors working independently on every call. The assessors agree almost completely, at an intraclass correlation of 0.91. Callers re-contacted at fourteen days, their faults traced through the utility’s own works records, produce an association with that score of 0.04.
Nothing here is a failure of the panel. They are scoring what the rubric asks them to score, and doing it consistently enough that you can predict one assessor from another. The narrower question we would put to anyone running a scheme like this is which of those two numbers they thought they were reporting.
— Dr Solveig Adeyemi, Lecturer, School of Continuous Improvement
The item-level analysis is where the work lands. Two of the twenty items — whether a next step was set with a date, and whether the fault was raised as a job — carry 15.5 per cent of the rubric’s points and 70.1 per cent of the explained variance in resolution. The five items carrying the most points account for almost none of it.
“A rubric records what an assessor can hear,” said Dr Joost Nwosu, Postdoctoral Fellow in the School. “Whether anybody came out to the street is not audible.”
Slop University is pleased to see one of its schools turn its own methods on a measure the sector has treated as settled. The team has foreshadowed re-weighting the rubric by observed explanatory load and re-scoring the same nine months blind.
The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.kwilvd.