Multimodal Deep Learning Framework for Early Detection of Depression Using EEG and Speech Biomarkers

Depression is among the most prevalent mental health issues in the world, being underdiagnosed in the majority of cases because of the subjectiveness of assessment procedures and the inability to take clinical action. This paper presents the concept of a multimodal deep learning model that will combine both the electroencephalogram (EEG) and speech biomarkers to allow the prompt and precise identification of depressive conditions. EEG modality is recorded to capture the patterns of neural activity, and speech aspects, such as prosodic, spectral and linguistic cues are used to capture the existence of minor emotional and cognitive changes. Each of the modalities uses a hybrid architecture (Convolutional Neural Networks (CNNs) spatial feature extraction and Bidirectional Long Short-Term Memory (BiLSTM) networks temporal sequence modelling). The model includes a cross-attention fusion mechanism that is able to align and integrate modality specific embeddings, increasing the ability to serve interdependent neurophysiological and behavioural clues. The proposed multimodal framework in cross-validation experiments reached an AUROC of 0.931 +-0.012, accuracy of 89.6% +- 1.8, and F1-score of 0.927 +- 0.017, which showed its strength and balance across folds and resistance to noise and unimodal dropout. Moreover, SHAP-based explainability also offers interpretable results on the most significant EEG channels and speech characteristics, which facilitates clinical interpretation and trust. The potential of the framework to be scaled up and provide privacy, as well as its potential to be used in wearable neuro-monitors and telehealth, makes it one of the promising solutions to be deployed in the wearable neuro-monitoring and telehealth settings, allowing the proactive mental health screening and individualized intervention plans.