Researchers at the School of Emergent Priorities have released new findings on the University’s library discovery portal, examining what happens to patron search outcomes when the system’s backend ranking model is upgraded but the search box patrons type into never changes.
Drawing on twenty months of search-session data spanning four consecutive upgrades to the portal’s ranking model, the team compared each release’s offline relevance score, assessed by library staff against a fixed panel of test queries, with how often a session ended in a patron actually opening a full-text item or placing a hold. The offline score nearly doubled across the study period; the rate at which searches ended in something found barely moved.
“This work opens important conversations about how much of a system’s improvement actually reaches the person searching it,” said Professor Verity Marris, Director of the Trajectory Analytics Group in the School of Emergent Priorities.
The release marks a significant milestone in the University’s ongoing work to understand its research and teaching systems from the inside out.
The team notes that a relevance-explanation feature intended to help close the gap did not shift outcomes measurably, a result they read as informative in its own right.
“We look forward to seeing where the library and information-science community takes these findings,” said Dr Petra Umbile, Postdoctoral Fellow in the School of Emergent Priorities.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.lwklbz.
Read the output: Findable, Not Found: Backend Relevance-Model Capability and the Retrieval Gap in the Library Discovery System (doi:10.5555/slop.lwklbz)