EEG and Speech Bimodal Fusion for Mental Disorder Recognition

Accurate depression grading is critical for clinical intervention. However, single-modal speech data suffers from class imbalance, and single-modal features fail to fully characterize depression states. To address these issues, this study proposes a depression grading scheme combining speech data enhancement and EEG-speech bimodal fusion. First, speech signals undergo preprocessing and spectral enhancement, with the SMOTE algorithm synthesizing minority samples to balance data distribution. Three models are then constructed: a speech CNN model for spectral feature extraction, an EEG Transformer model for capturing temporal neural features, and a cross-modal fusion model that associates bimodal features via attention mechanism. Controlled experiments with equal total samples and uniform grade distribution verify performance. Results show SMOTE increases the overall accuracy of the speech single-modal model by 17.8 percentage points, from 65.5% to 83.3%. Recognition accuracy for scarce grades, mild and moderate depression, rises by over 30 percentage points. The bimodal fusion model further improves accuracy by 6 percentage points, from 83.8% to 89.8%, compared with the optimized speech model, with all grades achieving stable accuracy above 88%. This study confirms SMOTE enhancement and cross-modal feature complementarity effectively improve depression grading accuracy, providing reliable technical support for clinical auxiliary diagnosis. Future work will expand multi-center datasets to enhance model generalization.