(link)

The paper compares actual brain signals from fMRI studies to the activation of neurons in an LLM. It finds that:

• Predictive coding theory offers an explanation to why LLMs can’t solve certain problems: while language models are optimized to predict nearby words, the human brain would continuously predict a hierarchy of representations that spans multiple timescales

• First, we confirmed that the activations of modern language models linearly map onto the brain responses to speech. Second, we showed that enhancing these algorithms with predictions that span multiple timescales improves this brain mapping. Finally, we showed that these predictions are organized hierarchically: frontoparietal cortices predict higher-level, longer-range and more contextual representations than temporal cortices.

Methodology:

• Make 300 people listen to short stories and record their brains through fMRI.

• We then fitted, for each voxel and each individual independently, a linear ridge regression to predict the fMRI signals from the activations of several deep language models.

• We then computed the corresponding ‘brain scores’ using held-out data, that is, the voxel-wise correlation between the fMRI signals and the predictions of the ridge regression input with the activations of a given language model

• Interesting: we first focused on the activations of the eighth layer of Generative Pre-trained Transformer 2 (GPT-2), a 12-layer causal deep neural network because it best predicts brain activity