Promoting Sign Language Awareness: A Deep Learning Web Application for Sign Language Recognition

Sign languages are essential for communication among deaf and hearing-impaired communities and with the hearing world. Developing tools for bidirectional translation between signed and spoken languages holds significant social value, promising to improve mutual understanding and reduce communication barriers. In this project, we aim to contribute to this area by developing a deep learning model for the task of isolated sign language recognition and deploying the model into an interactive web application, which can be used to aid in the annotation of continuous sign language videos, support the learning process for sign language students including hearing-impaired children and their families, and promoting broader sign language awareness. Our work in this project are twofold: 1) We constructed a LSTM (Long Short-Term Memory) model utilizing the dataset from the Google Isolated Sign Language Recognition competition hosted on Kaggle, which outperformed the baseline LSTM model with our feature engineering. We also reproduced a transformer based and another CNN based model. By comparing our model against the more complex transformer or CNN based model, we identified key factors contributing to performance improvements. 2) We deployed our model into a prototype web application designed with an interactive interface, aiming to make the model accessible for practical use. Through our work, we aspire to inspire further research interests of our students and increase broader awareness of sign language recognition fields. The project is open sourced here, and the web application is available here.