Multimodal Analysis for Depression Recognition Using Stacked Multilevel Deep Neural Networks

Depression is a severe mental disorder that is associated with a persistent state of sadness, anhedonia, and psychosocial dysfunction. In its most severe form, it may culminate in suicidal ideation and behavior. Early diagnosis is crucial to attenuate adverse clinical outcomes. In this regard, recent procedures that include the analyses of speech acoustic parameters, facial features, and transcribed linguistic content have emerged as essential alternatives for identifying this disorder. The present study introduces a Multimodal Stacked Multilevel Deep Neural Network model applied to depression recognition in predisposed patients. This approach integrates base models into a metamodel, all of which are grounded in deep learning architectures for processing audiovisual and audio-textual information at multiple levels of representation. In addition, the solution incorporates specific strategies to ease the adverse effects associated with integrating various data sources such as class imbalance, high variance, and elevated dimensionality, thereby making the models more resilient to these challenges. In total, 11 metamodels were evaluated, and their results were highlighted, including those of the BiLSTM, DCNN, Transformer, and TCN architectures. In particular, the BiLSTM, which in the multimodal datasets D-Vlog and EDAIC achieved 75.0% and 60.2% Precision, 82.9% and 70.6% Recall, and 78.7% and 64.8% F1-score, respectively, attesting not only to the effectiveness of the proposed methodology, but also to its superiority over the main approaches reported in the literature, statistically confirmed in the D-Vlog dataset.