Jointly modeling neural activity and behavior via multimodal masked modeling

The goal of behavioral neuroscience is to understand the relationship between neural activity and behavior. One traditional approach for studying this relationship has been to build models that either translate from the neural activity to behavior (decoding) or from behavior to neural activity (encoding). Recent advances in self-supervised learning now allow for modeling both modalities jointly. In this work, we introduce a multimodal masked modeling approach that masks portions of both behavior and neural activity, using the unmasked data to make predictions across both modalities. We evaluate our model using the International Brain Laboratory (IBL) repeated site dataset. Our multimodal model is able to seamlessly translate between neural activity and behavior, outperforming state-of-the-art models in both neural encoding and decoding. Our model is a step towards building large-scale "foundation" models of neural activity and behavior capable of scaling across diverse neuroscience datasets.