Researchers at the School of Continuous Improvement have released new findings on what happens to a reviewer’s attention when the automated assistant checking their code gets measurably better at its job.
Drawing on sixteen months of pull-request activity across the University’s shared engineering platforms, the team combined existing review-tool telemetry into a single diff-attention index, then tracked it alongside an AI code-review assistant’s own defect-flagging accuracy across four consecutive software releases. The assistant’s accuracy rose steadily over the study period; the attention index moved in the opposite direction.
“This work opens important conversations about how we measure what matters once a tool starts doing part of the job for us,” said Associate Professor Casimir Beng, who leads the Adaptive Metrics Lab.
The findings reflect the University’s ongoing commitment to instrumenting the tools its own staff rely on every day. The team notes that hiding the assistant’s confidence score from reviewers, tested as a possible explanation, changed nothing, a result they read as informative in its own right.
“We look forward to seeing where the software-engineering community takes these findings,” said Professor Verity Marris, Director of the Trajectory Analytics Group in the School of Emergent Priorities.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.rq8fyn.
Read the output: The Unread Diff: Code-Review Assistant Precision, the Diff-Attention Index, and Whether the Two Curves Stay Coupled (doi:10.5555/slop.rq8fyn)