In an effort to bring order to the Wild West of brain-activity AI, researchers have introduced NeuralBench—a unified framework for benchmarking models that analyze brain recordings. The first results from this large-scale effort show that so-called foundation models barely outperform task-specific ones, and that many practical challenges, like decoding thoughts or predicting clinical outcomes, remain far from solved.
The research
In a white paper published on arXiv, Hubert Banville and colleagues from institutions like Facebook AI Research and Université Paris-Saclay describe NeuralBench and its accompanying dataset, NeuralBench-EEG v1.0. The team aggregated 36 electroencephalography (EEG) tasks, 14 deep learning architectures, and 94 datasets, all accessed through a standardized interface. This is the largest benchmark of its kind for EEG-based models.
The findings are sobering. When pitted against each other, foundation models—large pre-trained models that can be fine-tuned for different tasks—only slightly outperformed models trained from scratch for specific tasks. Moreover, even the best models struggled with many tasks: cognitive decoding (e.g., reading mental states) and clinical predictions (e.g., detecting abnormalities) remained highly challenging. The authors note that these results highlight the need for more innovation, not just more data.
Why it matters
For the average person interested in their own brain, this research underscores that AI tools for brain analysis are still in their infancy. While they show promise in controlled settings, their reliability in real-world scenarios is limited. That means we should take claims of AI reading our minds or diagnosing conditions with a grain of salt. On the flip side, the push for standardized benchmarks like NeuralBench will accelerate progress, leading to better tools for understanding cognition and brain health in the future.
What you can do
You don't need an EEG to start understanding your brain. Online cognitive tests, like the ones at iqgenio, can give you a snapshot of your memory, attention, and reasoning skills. Tracking your performance over time can reveal patterns and help you target areas for improvement. And while AI won't read your thoughts just yet, you can use evidence-based brain training to sharpen your mind.
Source: arXiv q-bio.NC
Curious about your own brain? Take our free adaptive IQ test or try 306 brain training levels.