A paper from the School of Continuous Improvement finds an async stand-up bot's keyword-based blocker flag agreeing only slightly with an independent panel, with the blockers it misses taking four times as long to close as the ones it catches.
A single-site case study finds the bot's five-term keyword list agreeing with an independent panel only slightly, and its missed blockers taking four times as long to close
An automated blocker flag that a release-engineering team’s daily updates feed into caught one confirmed blocker in four that an independent review found. The other three took, on average, four times as long to close.
Nobody Typed The Blocker That Mattered, deposited today through the Office of Research Outputs, reports a single-site case study of the team’s move from a face-to-face morning stand-up to an async bot that collects a written status update and flags a blocker by keyword. Researchers in the School of Continuous Improvement set the bot’s flag against a two-person panel’s independent read of the same 402 updates, and against the team’s own record of how long each blocker sat open.
The panel found a genuine blocker in just under a third of updates; the bot’s flag caught fewer than half that share, agreeing with the panel at a level the researchers describe as slight. The finding folds into the University’s growing interest in what a workplace tool quietly stops doing once its predecessor retires. Widening the bot’s keyword list from five terms to twelve caught more of what the panel found and let through more of what it hadn’t, leaving the tool’s accuracy almost exactly where it started.
A room used to catch a blocker before anyone typed a word about it. We are still working out what a channel would need to do to catch it too.
— Dr Thandiwe Solberg, Senior Lecturer and Convenor, Demo Quarter
“The team’s stand-up was always worth watching once its shape changed,” said Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab. “What we can now say is which of its two jobs the new format actually kept.”
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.4e2id5.