A new Slop University research poster checks a national retail franchise's onboarding-video completion flag against player scrub telemetry, finding three in ten "completed" viewings land under a fifth of runtime actually watched.
Telemetry from 6,400 onboarding viewings finds three in ten "completed" videos left under a fifth of the runtime watched, worst after a policy check-in prompt
The Adaptive Metrics Lab keeps a standing brief: take an indicator an organisation already trusts, and work out what it would take to fool it. Its newest target sits nowhere near a dashboard. It’s the “completed” flag on a national franchise’s staff onboarding video, the one every new hire sits through before their first shift.
We didn’t need anyone to lie about watching it. We just needed the scrubber bar, which every video player already gives you for free.
— Dr Ingeborg Nwachukwu, Lecturer and Convenor, Capability Sprints
Scrubbed, Not Watched, deposited today through the Office of Research Outputs, pairs the completion flag from 6,400 onboarding viewings across 214 stores against the player’s own telemetry for the same sessions: dwell time, seek events, playback speed. Three in ten “completions” show under a fifth of the video’s runtime actually watched, the rest a single jump to the final frame. A platform-wide rollout of mid-video check-in prompts cut scrub-past rates across every onboarding segment but one: Workplace Conduct and Reporting, where scrubbing nearly doubled.
“The completion flag held at 97.6% through the whole redesign,” said Dr Dagny Okafor, Lecturer and Convenor, Evaluation of Evaluation. “A number that steady while the thing underneath it visibly changes shape is not a number I’d trust to manage anything on its own.”
The team’s next step drops the flag entirely in favour of scoring segment-level dwell time directly, starting, deliberately, with the segment everyone just learned to scrub past first.
The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.lsmiy7.