The Role of Neural Variability in Supporting Few-shot Generalization in Cortex

Cortical neurons exhibit a high degree of trial-to-trial variability in response to repeated presentations of the same stimulus. We propose that this variability serves a useful computational purpose: allowing the brain to generalize from a small number of examples. To formalize this theory, we consider a Gaussian model of few-shot classification, wherein we observe a few samples of each class, and each observation has trial-to-trial variability. Our model leads to three predictions about the optimal neural response variability that minimizes generalization error: (i) the optimal variability must have a covariance structure that is aligned with the covariance of each class; (ii) when focusing on two classes, the optimal covariance must be infinitesimal in the direction orthogonal to the class boundary; and (iii) the magnitude of the optimal variability must shrink with more samples (‘shots’) to learn from. To test these theoretical predictions, we analyze spiking neural activity from mouse visual cortex and find evidence consistent with each of these hypotheses: trial-to-trial variability is indeed aligned with each class’s covariance; the magnitude of variability shrinks in a task-specific direction with task engagement; and the magnitude of variability shrinks in all directions with increased stimulus familiarity. Finally, we test these ideas in a convolutional neural network (CNN), showing that injecting noise with the appropriate covariance structure at an intermediate layer can induce generalization over a new invariance space. Taken together, the data and simulations provide evidence consistent with the theory that cortical variability supports few-shot generalization.