A Dual-Mode Sign Language Translator Using LSTM Networks and NLP

The inability of the general population to engage with hearing-impaired individuals creates gaps which require assistive technologies to address these deaf barriers. This paper describes the implementation of a real-time two-directional sign language translator that uses Long Short-Term Memory (LSTM) networks for converting sign language to text and speech to sign language. For better accuracy, the WLASL (Word-Level American Sign Language) Dataset is employed to train the sign language recognition model. Input speech is acquired through a microphone using PyAudio API, and is recognized online with Google Speech API and offline with Sphinx. The resulting text is first processed through a context understanding Natural Language Processing (NLP) model and then is converted to American Sign Language using a dictionary-based machine translation model which displays static images or GIFs of the translated words. The translated words are shown in a simplified form ignoring complex grammar structures to put the user in an interactive environment. The static images or gif images are sponsors of signs that put the user in focus to A motion-centric user interface offers high usability, and was created with EasyGui. Under experimental results, it was proven the LSTM model was better than other models of detecting hand signs, and that the module translating speech into signs produced content in American Sign Language. The designed system offers an effective and scalable strategy to improve communication accessibility for the hearing-impairedpeople, aiding in the development of assistive technology as well as sign language translation research.