A new Slop University research poster sits with 96 householders as they answer one survey question about their smoke alarm, and finds the district rate that dispatches free-installation crews turning on which of six readings a respondent happens to apply.
96 in-home think-aloud interviews, and a district rate that falls 27 points once the program's own definition is read aloud
Do you have a working smoke alarm in your home? Ninety-six householders were asked exactly that, at their own kitchen tables, and invited to say aloud what they were thinking as they answered.
Working Means Six Things, deposited today through the Office of Research Outputs, reports what they said. Researchers in the School of Emergent Priorities coded the verbal reports into six readings of the word, none of them a variant of the others, and the two most common of them — the alarm is on the ceiling, and it sounded once during cooking — involve no test of the device at all. Ninety point seven per cent of the sample answered yes as asked. Read the program’s own definition aloud and the figure is 63.1, with 29 of the 96 changing their answer in the room.
There is a version of this result that reads as carelessness, and it is the wrong one. Every reading we coded is a reasonable thing to mean by the word. The question simply never said which one it wanted.
— Dr Ronja Oyelaran, Senior Lecturer, School of Emergent Priorities
Interviews ran for fourteen weeks across eight districts, twelve households in each, always in the room where the alarm is installed. Two coders worked blind to district on a frame fixed before the first visit. Because the rate is published by district and ranks them for free-installation effort, the team also re-ordered the eight on both readings: every district falls by at least 22 points, and the two ranked lowest as asked place fourth and sixth once the definition is applied.
Dr Ingrid Vasseur, Lecturer and Convenor of Futures in Committee, said the timing was the useful part. The next survey wave “is still being drafted”, she said, “and a question that can be read six ways is a great deal easier to fix before it has another year of trend sitting behind it”.
The School has lately taken to examining instruments that other bodies are about to reuse, and the University is content to see that become a habit.
The full poster is available from the University’s research repository under an open licence, doi:10.5555/slop.eusqt9.