Design and development of an AI-powered hand gesture recognition system for touchless screen interaction

Khamis, Shakiroh and Ahmad, Mazida and Idris, Aizal Yusrina and Ishak, Zurida (2026) Design and development of an AI-powered hand gesture recognition system for touchless screen interaction. In: 2026 7th International Conference on Artificial Intelligence and Data Science (AiDAS), 2-3 September 2026, Universiti Malaysia Sabah, Sabah.

PDF - Published Version Restricted to Repository staff only Download (1MB) | Request a copy

Abstract

The COVID-19 pandemic has accelerated the demand for touchless technologies that enhance public health and safety in shared environments. Hand gesture recognition offers an effective solution by enabling users to interact with digital devices without physical contact, thereby reducing the risk of contamination while improving the user experience. This paper presents the design and development of an AI-powered hand gesture recognition system for touchless screen interaction. The system was implemented in a Jupyter Notebook environment using OpenCV, MediaPipe, NumPy, and PyAutoGUI to perform real-time hand detection, gesture recognition, and virtual mouse control through a standard camera. The proposed system supports functions such as cursor movement, clicking, text selection, and scrolling using predefined hand gestures. The system was evaluated through usability testing involving IT professionals and students using a structured questionnaire covering accessibility, functionality, familiarization time, user interaction, physical contact avoidance, and overall satisfaction. The results showed that all respondents were able to download and use the system successfully, while the majority found the hand gesture dictionary helpful for learning the system. Scrolling and mouse cursor control were identified as the most useful features, and respondents unanimously agreed that the system effectively eliminated the need for physical contact. Overall, 61.9% of respondents rated the system 4 out of 5, indicating a high level of user satisfaction. The findings demonstrate that the proposed system provides an accessible, practical, and hygienic alternative to conventional input devices while promoting more natural human-computer interaction. Future enhancements will focus on expanding gesture functionalities and further improving system accuracy and usability.

Item Type:	Proceeding Paper (Other)
Uncontrolled Keywords:	Hand Gestures, Touchless screen, Human Computer Interaction
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Information and Communication Technology Kulliyyah of Information and Communication Technology Kulliyyah of Information and Communication Technology > Department of Computer Science Kulliyyah of Information and Communication Technology > Department of Computer Science
Depositing User:	Dr. Aizal Yusrina Idris
Date Deposited:	15 Oct 2026 15:09
Last Update:	15 Oct 2026 15:09
Queue Number:	2026-10-Q5617
URI:	http://irep.iium.edu.my/id/eprint/131953
Indexed In:	Google Scholar

Actions (login required)

View Item