EEG-based AI models that claim to detect clinical conditions may actually be picking up on who you are, not what your brain is doing. New research introduces the 'Identity Trap' and a toolkit to spot it.
The Research
A team led by Jun-You Lin from the University of California, San Diego, including Ying Choon Wu and Tzyy-Ping Jung, audited three popular EEG foundation models: LaBraM, CBraMod, and REVE. They tested them across four datasets, each arranged in a 2x2 layout based on whether the label (e.g., patient vs. control) varied within or between subjects, and whether a known cross-subject EEG marker existed.
Using their new diagnostic protocol, FMScope, they found that frozen (untrained) model representations contained 13 to 89 times more subject-identity variance than expected by chance — in all 12 model-dataset combinations. Fine-tuning made it worse, boosting identity variance by 10 to 63 percentage points. When they removed the subject-identity axis, label decoding accuracy improved where the label varied within a subject: by 6 to 12 percentage points in primary datasets, and up to 27 points in external cohorts.
They also discovered that aperiodic (1/f) brain activity is one carrier of subject identity: removing it reduced subject-probe accuracy by 9 to 19 points on two models. However, REVE saturated subject identity without relying on aperiodic signals.
Why It Matters
For anyone interested in EEG-based diagnostics or brain training, this means that high accuracy scores on clinical tests can be misleading. The models may learn shortcuts — using your unique brain signature (like a fingerprint) rather than the actual condition. That's why subject-disjoint cross-validation, though common, is not enough to ensure the model is learning genuine biomarkers.
What You Can Do
When you see claims about EEG AI accuracy, ask whether the researchers checked for identity leakage. Look for studies that use diagnostics like FMScope to separate subject identity from true signal. For your own cognitive training, remember that your brain's uniqueness is a feature, but it shouldn't mask real changes you want to track.
Source: arXiv q-bio.NC
Curious about your own brain? Take our free adaptive IQ test or try 306 brain training levels.
