Skip to main content
Researchers put a price on the study a systematic review misses

Researchers put a price on the study a systematic review misses

A new poster from the School of Emergent Priorities asks when a review team has read enough abstracts, and how much the answer depends on what a missed study is worth.

A stopping rule for abstract screening, replayed across fourteen completed reviews, barely moves when the price does

The School of Emergent Priorities has long taken an interest in the moment an institution decides it has done enough. Its newest research poster finds that moment somewhere unglamorous: the long middle of a systematic review, when a team is still reading abstracts and wondering whether the next hundred will turn up anything at all.

The Price of a Missed Study is led by doctoral candidate Ms Ylva Onyekwere with fellow candidate Mr Halvard Nnamani and Dr Runa Adegoke, Senior Lecturer and Convenor of the Strategic Drift Survey. Before any screening was revisited, each review team was asked how many hours it would spend to avoid missing one eligible study. The answers ranged forty-fold. Fourteen completed reviews were then replayed under a rule that stops once the next batch is no longer worth its reading time, and the spread in those prices moved the stopping point by a median five percentage points.

Every team we asked had a figure ready, and no two figures agreed. We owe them thanks for letting us test their answers against their own old decisions, and they can take some comfort from how little the disagreement turned out to cost.

— Mr Halvard Nnamani, Doctoral Candidate, School of Emergent Priorities

The work brings decision-theoretic framing, prioritised screening and a retrospective replay design into one account of how evidence synthesis reaches its natural end. The University counts it among a growing body of work on deciding well with less than everything in hand.

Dr Adegoke described the poster as “a question the School has been circling for some time, now put in a form other review teams can pick up and use.”

The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.8itaj5.