Advanced multimodal emotion recognition for Javanese language using deep learning

Arifin, Fatchul and Nasuha, Aris and Priambodo, Ardy Seto and Winursito, Anggun and Gunawan, Teddy Surya (2024) Advanced multimodal emotion recognition for Javanese language using deep learning. Indonesian Journal of Electrical Engineering and Informatics (IJEEI), 12 (3). pp. 503-515. ISSN 2089-3272

Preview	PDF - Published Version Download (697kB) | Preview
Preview	PDF - Supplemental Material Download (145kB) | Preview

Abstract

This research develops a robust emotion recognition system for the Javanese language using multimodal audio and video datasets, addressing the limited advancements in emotion recognition specific to this language. Three models were explored to enhance emotional feature extraction: the SpectrogramImage Model (Model 1), which converts audio inputs into spectrogram images and integrates them with facial images for emotion labeling; the Convolutional-MFCC Model (Model 2), which leverages convolutional techniques for image processing and Mel-frequency cepstral coefficients for audio; and the Multimodal Feature-Extraction Model (Model 3), which independently processes video and audio features before integrating them for emotion recognition. Comparative analysis shows that the Multimodal Feature-Extraction Model achieves the highest accuracy of 93%, surpassing the Convolutional-MFCC Model at 85% and the Spectrogram-Image Model at 71%. These findings demonstrate that effective multimodal integration, mainly through separate feature extraction, significantly enhances emotion recognition accuracy. This research improves communication systems and offers deeper insights into Javanese emotional expressions, with potential applications in human-computer interaction, healthcare, and cultural studies. Additionally, it contributes to the advancement of sophisticated emotion recognition technologies.

Item Type:	Article (Journal)
Additional Information:	External collaboration with UNY, Indonesia.
Uncontrolled Keywords:	Javanese emotion recognition; multimodal deep learning; audio-visual integration; emotion detection models; cultural emotion analysis; human-computer interaction
Subjects:	T Technology > TK Electrical engineering. Electronics Nuclear engineering > TK7800 Electronics. Computer engineering. Computer hardware. Photoelectronic devices > TK7885 Computer engineering
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Engineering > Department of Electrical and Computer Engineering Kulliyyah of Engineering
Depositing User:	Prof. Dr. Teddy Surya Gunawan
Date Deposited:	08 Oct 2024 09:05
Last Modified:	08 Oct 2024 09:05
URI:	http://irep.iium.edu.my/id/eprint/114892

Actions (login required)

View Item