A new study from researchers at IIIT Hyderabad, Microsoft, and other institutions reveals that instruction-tuned multimodal large language models (IT-MLLMs) show stronger alignment with human brain activity than other AI models. The study, uploaded to arXiv on June 9, 2025, uses fMRI data from people watching a naturalistic movie to compare how different AI systems represent information.
The Research
The team, led by Subba Reddy Oota and colleagues, analyzed brain responses from participants watching a movie with both video and audio. They extracted representations from six video and two audio IT-MLLMs across 13 different video task instructions. They then predicted fMRI activity from these AI representations. Results showed that instruction-tuned video MLLMs achieved roughly 9% higher brain alignment than in-context learning (ICL) multimodal models, 15% higher than non-instruction-tuned multimodal models, and 20% higher than unimodal baselines. Importantly, ICL models showed strong semantic organization (correlation r=0.78 with instruction text), while IT models showed weak coupling (r=0.14), indicating that IT models organize representations around functional task demands rather than surface semantics. Brain alignment varied across cortical regions, with task-conditioned subspaces emerging in higher-order areas.
Why It Matters
This research suggests that instruction tuning helps AI develop representations more similar to how the brain processes naturalistic stimuli. For cognitive science, it offers a new tool to map joint information processing in artificial and biological systems. For the average person, it underscores that context and task goals shape how information is encoded in the brain — a reminder that active, goal-oriented engagement may enhance learning and memory.
What You Can Do
To leverage this insight, practice active learning: set specific goals before watching a video or reading. This primes your brain to organize information more effectively, similar to how instruction tuning works in AI.
Source: arXiv q-bio.NC
Curious about your own brain? Take our free adaptive IQ test or try 306 brain training levels.
