Multimodal learning for deepfake detection using facial motion analysis and audio visual synchronization

Khalifa, Othman Omran and Nik Robidin, Nik Nurul Zulaikha and Hassan Abdalla Hashim, Aisha (2026) Multimodal learning for deepfake detection using facial motion analysis and audio visual synchronization. In: 2026 10th International Conference on Mechatronics Engineering (ICOM), 8-9 September 2026, Kuala Lumpur.

PDF - Published Version Restricted to Registered users only Download (1MB) | Request a copy

Abstract

Videos generated using advanced artificial intelligence-based techniques for deepfakes have become increasingly realistic and pose serious threats to media authenticity, cybersecurity, and trust. Methods used for conventional detection of deepfakes mostly based on visual features are often unable to detect new deepfakes which have little visual distortion. In this paper, we propose a multimodal deepfake video detection model that makes use of analysis of temporal facial motion and audio-video synchronization consistency. Our model incorporates a combination of Mobilenetv2 for feature extraction of visual features and TCNBiLSTM-Attention mechanism for feature extraction of temporal features. We extract audio-features using MFCC algorithm and measure synchronization consistency between speech and facial features by Dynamic Time Warping (DTW). We combine visual features, temporal features and audio features to perform classification to determine whether or not videos are manipulated. Experiments were performed on the FakeAVCeleb v1.2 dataset with balanced training and testing sets containing 1,000 videos each. Results obtained include 93.50% accuracy, 98.88% precision, 88% recall rate, 93.12% F1-score and an AUC of 98.16%

Item Type:	Proceeding Paper (Plenary Papers)
Uncontrolled Keywords:	Deepfake Detection, Multimodal Learning, Audio-Visual Synchronization, Temporal Facial Analysis, Dynamic Time Warping (DTW), Mel-Frequency Cepstral Coefficients (MFCC).
Subjects:	T Technology > T Technology (General) T Technology > TK Electrical engineering. Electronics Nuclear engineering
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Engineering > Department of Electrical and Computer Engineering Kulliyyah of Engineering
Depositing User:	Prof. Dr Othman O. Khalifa
Date Deposited:	10 Nov 2026 15:59
Last Update:	10 Nov 2026 15:59
Queue Number:	2026-11-Q6085
URI:	http://irep.iium.edu.my/id/eprint/132540
Indexed In:	UNSPECIFIED

Actions (login required)

View Item