A new paper from the School of Continuous Improvement finds that a small Spanish language model separates the future that predicts from the future that guesses at its middle layers, and that once the adverbs are taken away it does so about as well as the people who annotated its data.
A small model tells the conjectural future from the temporal one in its middle layers, about as well as its annotators could
Spanish has a way of saying “she must be at home” that looks exactly like “she will be at home”. A new paper from the School of Continuous Improvement asks where, inside a language model, the difference between the two is kept.
Estará en casa is by Dr Renke Sabel, Senior Lecturer and Convenor of the Indicator Commons, and Dr Dagny Okafor, Lecturer and Convenor of Evaluation of Evaluation.
An indicator is a promise about the future, and the School has spent a long time learning how many of its promises were really guesses. It has been a pleasure to spend a year with a language that keeps the difference in the verb.
— Dr Renke Sabel, Senior Lecturer and Convenor, Indicator Commons, School of Continuous Improvement
Working from more than two thousand sentences spoken by callers to radio phone-ins, the authors bring layer-by-layer diagnostic methods into dialogue with the study of Spanish grammar, aligning what a model holds at each stage of its reading with what trained listeners hear. The distinction emerges midway through the model’s depth, and when the adverbs that usually give the game away are removed, the model and its human readers find the same sentences difficult.
Dr Okafor credited the work’s foundations to colleagues outside the author list. “We owe the most to the three readers who went through every sentence three times,” she said, “and who were generous enough to disagree with one another whenever a sentence gave them reason to.”
The paper brings a School best known for its dashboards into the company of the University’s language research, a meeting the University regards as overdue and welcome.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.egi43w.