A School of Underlying Conditions paper suppresses the activity of small neural networks partway through training and reports that the lasting loss of accuracy is produced at the moment the suppression is lifted.
A thousand training runs in the School of Underlying Conditions vary when, how deeply and for how long a network is suppressed
One thousand and twenty small neural networks have been trained in the School of Underlying Conditions, most of them interrupted partway through, and the School has now reported what became of them. Putting a Network Under: Timing, Dose and Division of Transient Activation Suppression in Early Training is the work of Dr Brynjar Obiora, Senior Research Fellow, with Associate Professor Ragnhild Uzoma, Lead of the Basement Relations Group, and Dr Vigdis Emenike, Senior Lecturer.
We expect to go on switching things off and on again for some years, and to become steadily more careful about the second part.
— Dr Brynjar Obiora, Senior Research Fellow
The paper aligns a long-established laboratory protocol with the training of a small network. The activity of every unit is turned down for a set period early in training and then restored, with the timing, depth and length of the interruption varied in turn and the network examined long afterwards. An interruption of eight training epochs at half strength cost up to 7.4 points of final accuracy. The authors anchor that loss to the instant of restoration, and report that it largely disappears when the restoration is spread over two epochs.
Slop University welcomes the paper as a sign that its Schools will borrow an experiment as readily as an instrument, and return it in good order. Dr Emenike said the School had “come to regard endings as a subject in their own right, and will be allowing more time for them”.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.5gw0bw.