A new Slop University paper follows a home-repair platform's staggered switch to a fourteen-day satisfaction prompt, finding ratings fall by more than half a point on a five-point scale while the platform's own ninety-day repeat-callback record does not move.
A staggered platform rollout lets researchers separate what a satisfaction prompt measures from whether the job actually held
A national home-repair platform moved one text message back by a fortnight, and Slop University used the gap to ask what that message had been measuring all along.
The paper, from the School of Continuous Improvement, follows the platform’s region-by-region switch of its post-job satisfaction prompt from the minute a tradesperson marks a job complete to fourteen days afterwards. Across 38,714 jobs in nine metropolitan regions, the delayed prompt pulled the average rating down by 0.55 points on a five-point scale and split the distribution in two, with one-star responses rising from under two per cent to ten. The platform’s own record of whether the job held — a repeat callback to the same address for the same fault inside ninety days — did not move.
A tap that is dry when the van leaves is not yet a tap that holds. We were not testing whether households answer honestly. We were asking whether anyone had given them time to find out.
— Dr Mirela Hanke, Postdoctoral Fellow, School of Continuous Improvement
Only the delayed rating turned out to bear any relationship to the callback record. The immediate one, which the platform’s ranking of tradespeople leans on, bore none.
“Timing is the part of an instrument nobody puts in the specification,” said Dr Marit Osayande, Lecturer and Convenor of the Living Dashboard. “It is usually the part doing the most work.” The University counts the result among the more useful things a routine operational change has handed its evaluation-ecosystem agenda.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.6fso51.