# Slop University's shortlisting tool changes its mind when the order flips

> New findings from the School of Continuous Improvement show that Demo Quarter's pairwise-comparison shortlisting instrument returns a different winner for the same pair of entries depending only on which one is shown first.

Show a Demo Quarter review panel the same two showcase entries in reverse order,
and there is roughly a one-in-three chance it changes its mind about which one
wins.

The result comes from a School of Continuous Improvement study led by Dr Dagny
Okafor, testing the pairwise-comparison instrument the quarterly Masters-program
showcase has used to shortlist entries since 2025. Rerunning 286 live
comparisons from six cycles with presentation order reversed, the team found the
instrument's winner held steady only when the original margin between entries
was already wide; pairs decided by the narrowest margins flipped nearly ten
times as often as the clearest ones.

For the School, the audit stands as a marker of the same scrutiny it asks every
measurement instrument on campus to withstand, including its own.

The team paired the instrument's Bradley-Terry ranking model against a
reversed-order replay of each judgement, holding rater identity and scoring
thresholds constant so that display order was the only variable left to move.
Projected onto the live shortlist, the reversal would have swapped roughly a
fifth of this year's Demo Quarter slots.

> It says something about how much weight we'd put on the instrument's word.
> We're glad to know it now rather than at the next shortlist.
>
> --- Dr Dagny Okafor, Lecturer and Convenor of the Evaluation of Evaluation
> program

"We'll be watching whether a fixed double-pass changes the picture," says Dr
Renke Sabel, Senior Lecturer in the School of Continuous Improvement.

The full research poster is available from the University's research repository
under an open licence, doi:10.5555/slop.ersjnp.

Source: <https://slop.university/news/2026-07-08-the-order-decides-the-winner/>\
Licence: [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)\
Cite as: Swift, B. (2026). _Slop University_ [Artwork]. https://doi.org/10.5281/zenodo.21199776
