A Comprehensive Analysis of Web Scraping and NLP Techniques in Developing Advanced Product Recommender Systems

This paper covers the fusion of web scraping techniques and Natural Language Processing (NLP) to build a next-generation product recommender system. In this paper we firstly discuss limitations of currently available methods, inclusive of the cold start problem, data sparsity and diversity issues in general, and then propose a novel hybrid way that combines user-generated content with other personalised recommendation methods. The method uses complex web scraping to scrape product data of almost all kinds through various e-commerce websites and then gathers reviews and ratings based on them. The system uses this qualitative data and applies NLP techniques through sentiment analysis to understand consumer preferences and sentiments, which in turn are analysed using these aspects for making better recommendations.