EXPLAINABLE MULTIMODAL AI FOR FAIR INTERVIEW ASSESSMENT: AN ADAPTIVE DEEP LEARNING APPROACH WITH COMPREHENSIVE XAI

With the increased adoption of AI-driven interview systems, concerns about fairness, bias, and interpretability have gained significant importance. This work introduces an XAI-based multimodal evaluation system that integrates attention-based neural networks with SHAP and LIME explanations for transparent decision-making. The proposed hybrid architecture analyzes candidate gestures, vocal tone, and linguistic responses with CNNs, MFCC-based audio embeddings, and DistilBERT transformer encoders. It leverages cross-modal attention fusion, Sharpness-Aware Minimization (SAM) optimization, and generative synthetic data augmentation to address challenges in data scarcity and bias. The work ensures comprehensive explainability by integrating SHAP, LIME, attention visualization, and Grad-CAM techniques. The experimental results on CMU-MOSI and synthetic datasets yield state-of-the-art performances with 87.3% accuracy and 0.91 ROC-AUC, ensuring fairness with a demographic parity difference of 0.03 and improving recruiter trust by +44%. Model outputs provide not only predictions of confidence, adaptability, and sentiment but also interpretable visualizations that highlight which modality contributed most to decisions, thus enhancing recruiter trust while reducing algorithmic opacity