A year-long deployment of an automated sunset scorer across the School of Continuous Improvement's Indicator Commons finds retirement the least common outcome for anything it flags.
A year of flagging across the Indicator Commons ends in four retirements, once the scorer's own uptime joined the Living Dashboard
Slop University’s Indicator Commons holds 214 active indicators. A year ago, the Adaptive Metrics Lab built a tool whose entire purpose was to make that number smaller: a sunset scorer that flags stale, redundant, or rarely-queried indicators for the registry’s standing review committee to retire.
Over twelve months the scorer flagged 138 indicators, nearly two-thirds of the registry, as retirement candidates. Four were retired. Most of the rest were redefined, retained with a written justification, or are still waiting on the committee. The School of Continuous Improvement traces the gap to the scorer’s own uptime, added to the Living Dashboard two months into the deployment: once the instrument’s continued operation became something the University could see, keeping it running began to compete for attention with acting on what it found.
The School reads the result less as a defect in the scorer than as evidence of what happens once an instrument’s own continuity becomes visible on the same dashboard as the things it measures.
We built the scorer to answer one question, and it answered it. What it also did was give the registry something new to keep an eye on.
— Dr Renke Sabel, Senior Lecturer in the School of Continuous Improvement
“Every instrument we build eventually needs one to watch it,” said Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab. “We hadn’t expected to need it quite this soon.”
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.03ley4.