Dual-language visual-assisted speech command recognition system for hearing-impaired learners during driving lessons

Alif Roslan, Muhammad and Samad, Rosdiyana and Noordin, Nurul Hazlina and Abdullah, Nor Rul Hasma and Mahfuzah, Mahfuzah and Pebrianti, Dwi (2026) Dual-language visual-assisted speech command recognition system for hearing-impaired learners during driving lessons. In: 2026 IEEE International conference on Mechantronics, 8 September 2026, Kulliyyah of Engineering, IIUM.

PDF - Published Version Restricted to Repository staff only Download (1MB) | Request a copy

Abstract

Effective communication between driving instructors and hearing-impaired learners is essential to ensure safe and effective driving lessons. Conventional verbal instructions may not be accessible to hearing-impaired learners, while sign language or hand gestures may require learners to shift their visual attention away from the road. This study presents a dual-language visual-assisted speech command recognition system designed to convert driving instructions into visual feedback for hearing-impaired learners. The proposed system uses Malay and English driving command datasets, where audio signals are pre-processed, converted into spectrogram representations, and classified using a convolutional neural network model. Several model configurations were evaluated by varying the optimizer, batch size, and network depth. Based on the experimental results, the English dataset achieved the highest average accuracy of 88.45%, followed by the raw Malay dataset with 85.23% and the augmented Malay dataset with 81.90%. Although the augmented Malay dataset produced lower accuracy, it provides greater variability by including noise and speech variations, making it more representative of real-world driving environments. A graphical user interface was also developed to display the predicted driving instruction using text and visual cues. The proposed system demonstrates the potential of bilingual speech command recognition and visual feedback to support safer and more inclusive driving instruction for hearing-impaired learners.

Item Type:	Proceeding Paper (Invited Papers)
Uncontrolled Keywords:	assistive technology, convolutional neural network, driving lesson, hearing-impaired learners, speech command recognition, spectrogram, visual feedback
Subjects:	T Technology > T Technology (General) T Technology > T Technology (General) > T10.5 Communication of technical information
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Engineering > Department of Mechanical Engineering
Depositing User:	Dr Dwi Pebrianti
Date Deposited:	02 Nov 2026 09:11
Last Update:	02 Nov 2026 09:11
Queue Number:	2026-10-Q5922
URI:	http://irep.iium.edu.my/id/eprint/132350
Indexed In:	Google Scholar

Actions (login required)

View Item