A Novel Self-Supervised Learning Based on Masked Signal Modeling for Multimodal Physiological Emotion Recognition

Multimodal physiological emotion recognition is an essential and challenging field of human–computer interaction (HCI). However, the collection of labeled physiological signals is difficult and expensive, especially for electroencephalogram (EEG). To address the above issue, we propose a novel self-supervised learning (SSL) method, named the Masked-signal Modeling-based Self-supervised Learning (MMSL), to train model using numerous unlabeled physiological signal datasets. To capture robust and effective general representations, masked-signal modeling (MSM) is applied to pretrain within each modality through reconstructing EEG and electrocardiogram (ECG) signals by masking a certain proportion of signal. The pretrained signal encoder is then transferred to the SSL phase. In the SSL phase, representation alignment (RA) and cross-modal transformer (CMT) module are designed to capture effective representations without manual labeling through several signal transformations. A multimodal tensor fusion (MTF) technique is then used to integrate abundant multimodal information for transformation recognition. To verify the effectiveness of the proposed method, the pretrained modules are either frozen or fine-tuned for emotion recognition on the AMIGOS dataset. Experimental results demonstrate that our method achieves superior performance in arousal and valence levels compared to recent studies. Furthermore, several visualized results illustrate that the proposed method is capable of recognizing emotion states.