Facial Emotion Recognition with Sentiment Analysis for Cross-Dataset Generalization

Facial Emotion Recognition (FER) is fundamental to affect-aware intelligent systems, providing important services and benefits to healthcare, education, and Human-Computer Interactions (HCI); however, current FER methodologies frequently struggle to generalize across both controlled and uncontrolled environments due to the factors of pose, lighting, occlusion, and imbalance of emotion classes in the dataset being utilized. This article proposes a deep learning-based FER system that includes attention-guided feature extraction, thereby allowing the FER system to emphasize the most discriminative areas of the face while increasing the system's capability for recognizing facial expressions. The FER system is assessed using both the JAFFE and the AffectNet datasets representing controlled laboratory conditions compared to more naturalistic conditions, respectively; in contrast to developing an emotion classification system alone, the authors offer another level of emotion categorization beyond “emotional” by establishing a higher-order classification of emotions into “negative,” “neutral,” or “positive” affective states to enhance the interpretability of the FER system's results. The experimental results demonstrate strong FER performance, as achieved through accurate classifications of 98.2% on JAFFE and 72.4% on AffectNet, in addition to indicating the stable convergence of the FER system and balanced predictability of the FER system, as indicated by average precision, recall, and F1-scores of 71.8%, 72.1%, and 71.9%, respectively, suggesting the overall capacity for significant generalization of the FER system across all environments.