A School of Emergent Priorities paper follows 6,431 rehomings across three centres and finds a placement-suitability score's apparent accuracy is carried by the listing queue the score itself sets, disappearing in the weeks when building works set the order instead.
Among the 604 dogs listed in block-emptying order during a rolling kennel refurbishment, the score's discrimination fell to a level not separable from chance
A building programme did what nobody had thought to design. Accurate Only in Order, published today by the School of Emergent Priorities, uses the weeks in which one rehoming service could not list its animals in its usual order to ask what its placement-suitability score had been measuring all along.
The service scores every dog on a fifteen-item schedule on the third day after intake. The result prints the four suitability fields on the kennel-door card, and the same composite sets the order in which animals are photographed, written up, and published to the adoption site. Across fifty-four months and 6,431 listings the score separated the dogs later returned from those not returned with a c-statistic of 0.708, close to the figure the service reports each year. Among the 604 dogs listed while their blocks were being rebuilt, and therefore in the order the works emptied them, the same score reached 0.528.
The refurbishment was an inconvenience for everybody involved, and it is the only reason we can say anything at all. We are grateful the records were kept as carefully through it as they were before.
— Dr Iben Chikere, Lecturer, School of Emergent Priorities
Professor Verity Marris, Director of the Trajectory Analytics Group, placed the work in “a longer conversation the School keeps having with itself about what a forecast is actually for”, and noted that the service had been “generous with records that were never assembled with a question like this in mind”.
The paper also reports what the score does order reliably: the wait. Median days from assessment to first listing ran from 3 in the top scoring band to 23 in the bottom, and adjusting for that wait leaves the score’s association with return indistinguishable from none. The work was supported by two awards under the Anticipatory Capability Fund.
The release adds to a body of work the University has been assembling on instruments asked to forecast and to schedule at the same time. The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.203dz3.