A new study finds that certain layers of OpenAI's Whisper speech AI correspond closely to how the human brain processes speech. The research, presented at the ICLR 2026 Workshop on Representational Alignment, shows that intermediate layers—not the first or last—provide the strongest match with intracranial brain recordings.
The Research
Matteo Ciferri and colleagues (University of Rome, Harvard Medical School) recorded electrocorticography (ECoG) from 12 epilepsy patients listening to natural speech. ECoG uses electrodes placed directly on the brain, giving millisecond-precision data. They then fed the same speech into OpenAI's Whisper, a deep neural network trained on 680,000 hours of multilingual audio.
To compare Whisper's internal representations to the brain signals, the team developed a time-resolved neural encoder that combined Whisper embeddings with a recurrent temporal model and soft attention. This allowed them to examine layer-by-layer how well each of Whisper's 32 layers predicted neural activity. The middle layers (around layers 15-20) showed the highest correspondence, supporting a hierarchical alignment between the model's processing stages and cortical speech processing.
When compared to simpler linear models using the same speech features, the temporally structured encoder improved prediction accuracy by 15-20%. Attention maps revealed that the model focused on specific time points in the speech stream to predict neural responses, aligning with known temporal dynamics of speech perception. A phonemic analysis further showed that electrodes informative for encoding formed clusters corresponding to phoneme categories (like consonants vs. vowels), consistent with known functional organization of auditory cortex.
Why It Matters
This research suggests that deep learning models like Whisper can serve as a useful framework for understanding how the brain represents speech in real time. The hierarchical match implies that both AI and human brains process speech in stages, from simple acoustic features to complex linguistic abstractions. For anyone curious about their own cognition, this reinforces that speech perception is a dynamic, multi-layered process—not a single snapshot.
What You Can Do
To support your brain's speech processing, try active listening: focus on one speaker in a noisy environment, paraphrase what they said, and notice the distinct sounds (phonemes) of words. Regular practice may sharpen your auditory cortex's hierarchical analysis.
Source: arXiv q-bio.NC
Curious about your own brain? Take our free adaptive IQ test or try 306 brain training levels.
