A new paper from the School of Emergent Priorities audits the campus welcome desk's self-logged "resolved on first ask" rate against an independent same-day survey, finding the two measures share almost no signal even after a script revision meant to close the gap.
A same-day survey of 618 enquirers finds the desk's resolved-on-first-ask flag little better than chance at predicting the outcome
Ask Slop University’s welcome desk a question this term, and student ambassadors are trained to give a confident, specific answer on the spot. Whether that answer actually sends an enquirer where they need to go is a different question, and until now it was one the desk’s own performance figures had never been asked to answer. A new paper from the School of Emergent Priorities finds the desk’s self-logged “resolved on first ask” rate shares almost no signal with an independent survey of where enquirers actually ended up.
The desk’s confident-answer policy took effect at the start of the current term, and its resolution log has since served as Student Services’ headline account of how well the service performs. Researchers from the School of Emergent Priorities set that log against a same-day survey of 618 enquirers sampled across the term, an audit the School frames as part of its wider habit of checking what an institution’s own instruments actually capture before anyone leans on them further.
A confident answer and a correct answer aren’t the same claim, and the desk’s own log has been treating them as if they were.
— Dr Lindiwe Achterberg, Lecturer, School of Emergent Priorities
“The survey wasn’t designed to catch anyone out,” said Dr Anneke Tolan, Senior Research Fellow. “It was designed to see whether the number Student Services already reports is measuring what it claims to.”
Over fourteen weeks, the team compared the desk’s own resolution flag against enquirers’ same-day accounts of where they ended up, tracking the gap across a policy change midway through the term that asked ambassadors to escalate uncertain questions rather than guess. The gap held steady regardless.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.1jtnco.