A School of Continuous Improvement poster tests whether the University's mandatory AI-literacy sign-off predicts what research students can actually do, and finds self-rated fluency and measured task scores almost unrelated.
A validation study across a full postgraduate cohort finds the mandatory self-assessment barely tracks measured task performance
How a research student rates their own command of the University’s AI tools, and how they perform when actually set a task with them, are two different measurements. A new poster from the School of Continuous Improvement reports that, across a full postgraduate cohort, the two barely move together.
The study set a single intake a standardised battery of tool tasks, scored without reference to how confidently each student had rated themselves after the mandatory AI-literacy module, and compared the results. Across 118 students, self-rated fluency accounted for under two per cent of the variation in measured task scores; the certification threshold, the team reports, admitted strong and weak performers in much the same proportion. What the self-assessment separated cleanly was not task score but how quickly a student had finished the module.
The University presents the work as an instance of the scrutiny it asks of its own instruments. “A measure everyone trusts and no one has checked against performance is exactly the kind of instrument our programme exists to revisit,” said Dr Dagny Okafor, who convenes the School’s Evaluation of Evaluation programme. “The point is not that the sign-off is wrong. It answers a narrower question than the one we have been asking of it.”
Dr Renke Sabel, whose work catalogues how indicators persist once adopted, said the certificate had settled into the induction dashboard “the way most indicators do: quietly, and without ever being asked to prove itself twice.” The School frames the finding within its standing programme on measures that outlive the evidence for them, and notes that a task-anchored re-validation would be a reasonable next step before the sign-off is read as a marker of readiness.
The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.2rdi9d.