New findings from the School of Emergent Priorities test whether a capability-practice pattern found in two earlier studies reappears in a third setting, and trace what became of the result once it reached the School of Continuous Improvement's own dashboard.
A capability-practice pattern fails to reappear in a third setting, then takes on a life of its own in the dashboard
A drafting tool the University built to help sessional markers write assignment feedback has cleared several months of testing without producing the one effect researchers went looking for: any sign that its rising capability was changing how markers do the job.
The result answers a question the School of Emergent Priorities set out from two earlier University findings, in lecture-capture scheduling and meeting-minute generation, each linking a system’s growing capability to a dip in the practice it assisted. Researchers tracked the feedback tool’s capability climb against a specificity measure coded from markers’ written comments, across three teaching periods and an opt-out cohort who never adopted the tool at all. Neither comparison turned up a relationship worth reporting on its own terms — which the team is reporting anyway, in full, as a finding rather than a footnote.
A second thread followed the null result into the School of Continuous Improvement’s own quarterly reporting, where the same figures were characterised, cycle over cycle, in steadily warmer terms than the numbers ever changed.
A finding that fails to repeat is still information. The interesting part turned out to be what the University did with it afterwards, not the replication itself.
— Professor Verity Marris, Director of the Trajectory Analytics Group in the School of Emergent Priorities
“Watching your own reporting cycle get read this closely is not entirely comfortable,” said Associate Professor Casimir Beng, Lead of the Adaptive Metrics Lab in the School of Continuous Improvement. “It is, at least, the kind of scrutiny the Living Dashboard exists to invite.”
The University treats the result as continuous with its longer-running effort to test its own conclusions before it leans on them a second time.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.vh4mkq.