Transformer based Multimodal Mental Health Assessment Framework Using Cross-Attention Mechanisms on Audio-Visual Data

This research work offer a novel multimodal Transformer framework for mental health assessment, which exploits cross-attention to bind audio and visual modalities. Audio signals undergo processing using wav2vec 2.0, thereby yielding rich acoustic features, whereas facial representations are derived from ResNet 50. Text reasons with BERT, using inputs from both the Bipolar Disorder Corpus and the Extended Distress Analysis Interview Corpus (E-DAIC). These modalityspecific embeddings then merge using a cross-attention Transformer block. Our model largely outperforms the typical baselines, raising the F1-score by 8.4% and binary distress classification AUC by 6.7% when compared to competing architectures of BLSTM and early-fusion CNN-LSTM. These results show the power of cross-attention in modeling essential inter-modal dependencies. This methodology is scalable, meaningfully interpretable, and brings in significant promise when deploying clinical mental health screening systems.