# Call quality scores track assessor agreement, not whether the fault was fixed

> A new Slop University research poster scores 384 recorded calls to a water utility's contact centre against the twenty-item rubric behind its service-quality figure, and finds near-total assessor agreement with no relationship to whether the caller's fault was resolved.

A caller reports a leak. Somewhere behind that call, three people who were not
on it will listen to the recording and agree, closely, about how it went.

_Agreement Without Resolution_, deposited today through the Office of Research
Outputs, asks what that agreement is worth. The School of Continuous Improvement
scored 384 recorded calls to one water utility's contact centre --- leaks, bills,
meter faults --- against the twenty-item rubric the centre reports its service
quality from, with three accredited assessors working independently on every
call. The assessors agree almost completely, at an intraclass correlation of
0.91. Callers re-contacted at fourteen days, their faults traced through the
utility's own works records, produce an association with that score of 0.04.

> Nothing here is a failure of the panel. They are scoring what the rubric asks
> them to score, and doing it consistently enough that you can predict one
> assessor from another. The narrower question we would put to anyone running a
> scheme like this is which of those two numbers they thought they were
> reporting.
>
> --- Dr Solveig Adeyemi, Lecturer, School of Continuous Improvement

The item-level analysis is where the work lands. Two of the twenty items ---
whether a next step was set with a date, and whether the fault was raised as a
job --- carry 15.5 per cent of the rubric's points and 70.1 per cent of the
explained variance in resolution. The five items carrying the most points
account for almost none of it.

"A rubric records what an assessor can hear," said Dr Joost Nwosu, Postdoctoral
Fellow in the School. "Whether anybody came out to the street is not audible."

Slop University is pleased to see one of its schools turn its own methods on a
measure the sector has treated as settled. The team has foreshadowed re-weighting
the rubric by observed explanatory load and re-scoring the same nine months
blind.

The full poster is available from the University's research repository under an
open licence, doi:10.5555/slop.kwilvd.

Source: <https://slop.university/news/2026-08-04-what-the-panel-agrees-about/>\
Licence: [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)\
Cite as: Swift, B. (2026). _Slop University_ [Artwork]. https://doi.org/10.5281/zenodo.21199776
