Deep learning-based detection of depression by fusing auditory, visual and textual clues.

BACKGROUND

Early detection of depression is crucial for implementing interventions. Deep learning-based computer vision (CV), semantic, and acoustic analysis have enabled the automated analysis of visual and auditory signals.

OBJECTIVE

We proposed an automated depression detection model based on artificial intelligence (AI) that combined visual, audio and text clues. Moreover, we validated the model's performance in multiple scenarios, including interviews with chatbot.

METHODS

A chatbot for depressive symptom inquiry powered by GPT-2.0 was developed. The Brief Affective Interview Task was designed as supplement. Audio-video and textual clues were captured during interview, and features of different modalities were fused using a network with a multi-head cross-attention mechanism. To validate the model's generalizability, we conducted external validation using an independent dataset.

RESULTS

(1)In the internal validation set (152 depression patients and 118 HCs),the multimodal model achieved good predictive power for predicting depression in all scenarios, with an area under the curve (AUC) over 0.950 and an accuracy over 0.930. Under the symptomatic interview by chatbot scenario, the model achieved exceptional performance, with an AUC of 0.999. Specificity decreases slightly (0.883) in the Brief Affective Interview Task. The multimodal model outperformed unimodal and bimodal counterparts.(2)For external validation under the symptomatic interview by chatbot scenario, a geographically distinct dataset (55 depression patients and 45 HCs) was employed. The multimodal fusion model achieved an AUC of 0.978, though all modality combinations exhibited reduced performance compared to internal validation.

LIMITATIONS

Longitudinal follow-up was not conducted in this study, and severe depression applicability requires further study.