Artificial Intelligence-Based System for Word Recognition and Extraction in Indonesian Sign Language Videos

The Indonesian Sign Language System (SIBI) is a standardized sign language for deaf and mute communities, particularly in formal education. However, current solutions usually use static images or translate only alphabets. Furthermore, studies involving words used in academic settings are limited. This study develops and evaluates a 3D Convolutional Neural Network (3D CNN) and a CNN-LSTM model to recognize SIBI gestures commonly used in academic settings from recorded video data. Data preprocessing was done using frame resizing, temporal sampling, and data augmentation techniques to enhance its variability and robustness. Experimental results show that the CNN-LSTM model consistently outperforms the 3D CNN, achieving accuracies and F1 scores above 0.95 across all dataset variations. However, the CNN-LSTM has slower inference (2.70 ms per frame) compared to the 3D CNN (0.74 ms per frame). On top of that, a real-time SIBI gesture recognition application is made using the CNN-LSTM model to evaluate its performance in real-life settings on different hardware configurations. These findings highlight the trade-off between accuracy and computational efficiency, suggesting CNN-LSTM is more suitable for accuracy-critical applications, while 3D CNN offers advantages for real-time deployment. Future research should expand the dataset scope and explore advanced model architectures to further improve recognition of SIBI words.