A Multimodal Deep Learning Framework for Product Recommendation Using ResNet-50, BERT, and User Behavior Modeling

E-commerce sites use Product Recommendation Systems to keep users engaged, increase the likelihood of users becoming satisfied customers, and convert extremely well. Traditional methods of creating a product recommendation system lack the ability to take advantage of the sparse nature of user-item interactions, as well as not having enough contextual information about the recommended products and the user's activity. The advances in artificial intelligence (AI), particularly in the areas of deep learning and Multimodal Fusion, have led to powerful combinations of both textual and visual inputs that improve prediction accuracy. For example, ResNet-50 is a vision encoder that accurately captures the visual characteristics of products, while BERT is a transformer-based language model that accurately captures the semantic relationships between text and products. In addition, Neural Collaborative Filtering (NCF) provides a robust framework to model user behavior data and supports Explainable AI (XAI) to provide stakeholders (including businesses) with insight into the decision-making process for product recommendations. In this paper, we will describe a Multimodal Deep Learning Framework that uses ResNet-50 (to encode products' images), BERT (to encode product descriptions and reviews) and NCF (to model user preference) for generating very accurate recommendations for products. Recommendations for consumers come in the form of a Top-N list (i.e. the most relevant products). This Top-N recommendation, based on the recommendation system, is created using an encoder-decoder model that combines the visual, text and behavior encodings into a common embedding space.SHAP-based explainability methods have been developed to create an average sample from all images, texts and behaviors that are important for the Top-N recommendation.Comparative studies show that the proposed hybrid multimodal approach produces better accuracy, precision and recall than standalone (single) mode or collaborative filtering using visual, text and behavior. This study provides the foundation for the integration of multiple sources (visual, text, and behavior) in a recommendation system.