Multimodal Real-Time Translation System for Hearing Impaired Accessibility Using Deep Learning

Hearing-impaired individuals often face significant communication challenges in environments that rely

on spoken language, impacting their social, educational, and professional interactions. This paper introduces a

multimodal, real-time translation system using deep learning to enhance accessibility for the hearing impaired by

providing seamless translation of spoken language into text and sign language. The system combines multiple

advanced models to achieve a fully integrated solution: a transformer-based model for speech-to-text conversion,

delivering 96.5% accuracy in diverse acoustic conditions; a CNN-based sign language recognition module, achieving

92.3% accuracy across varied gestures and hand configurations; and a sequence-to-sequence text-to-sign translation

model with a BLEU score of 88.7, generating expressive, animated sign language outputs. Our system operates with

an average latency of 150 milliseconds, ensuring minimal delays and real-time responsiveness suitable for interactive

applications.

Evaluation of the system in real-world scenarios, including classrooms, workplaces, and public spaces, demonstrates

its robustness, accuracy, and scalability, particularly when deployed on mobile devices and wearables. This

multimodal approach enables flexible, efficient communication, adapting dynamically to individual preferences and

settings. Future enhancements will focus on expanding the system’s capabilities to support multiple spoken languages

and regional sign languages, as well as integrating edge computing for privacy-preserving, on-device processing.

This work represents a significant step toward inclusive and accessible communication technologies, positioning

deep learning at the forefront of assistive solutions for the hearing impaired.