SwinAraBERT-Fusion: a multimodal transformer architecture for robust Arabic OCR and handwriting recognition with diacritic-aware contextual modeling

Eltayeb, Sarra and Khalifa, Othman Omran and Hassan Abdalla Hashim, Aisha (2026) SwinAraBERT-Fusion: a multimodal transformer architecture for robust Arabic OCR and handwriting recognition with diacritic-aware contextual modeling. In: 2026 10th International Conference on Mechatronics Engineering (ICOM), 8-9 September 2026, Kuala Lumpur.

PDF - Published Version Restricted to Registered users only Download (1MB) | Request a copy

Abstract

Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR) remain persistently challenging due to the script's inherently cursive nature, context-dependent positional letter forms, dense diacritic system (tashkeel), and high inter-class visual similarity among characters. While transformer-based architectures have driven significant advances in text recognition, most existing Arabic OCR systems rely on unimodal visual encoders that underutilize the rich morphological and syntactic structure of the Arabic language. This paper proposes SwinAraBERT-Fusion, a novel multimodal deep learning architecture that integrates a SwinV2 hierarchical vision transformer with an AraBERT contextual language model through a cross-attention fusion mechanism augmented with adaptive gating. The system is designed for line-level recognition of both printed and handwritten Arabic text, with explicit handling of diacritics. Training employs a two-stage curriculum learning strategy on combined benchmark datasets (KHATT, IFN/ENIT, APTI, SARD). Preliminary experiments on "Qalam_Net_V2" baseline, a fine-tuned Transformer-based architecture built on TrOCR model using 500 test samples yielded a Character Error Rate (CER) of 10.87% and Word Error Rate (WER) of 56.20%, revealing critical failure modes - diacritic substitution and short-word sensitivity - that directly motivate our multimodal design. The proposed architecture is expected to significantly outperform unimodal baselines by leveraging AraBERT's morphological priors to resolve ambiguities that visual features alone cannot disambiguate. Contributions include: (1) an open-source multimodal Arabic OCR/HWR framework; (2) empirical demonstration of improved CER/WER over state-of-the-art unimodal systems; and (3) comprehensive analysis of cross-modal fusion strategies for Arabic script recognition.

Item Type:	Proceeding Paper (Plenary Papers)
Uncontrolled Keywords:	Arabic OCR, Handwriting Recognition, SwinV2, AraBERT, Multimodal Fusion, Diacritics, Transformers
Subjects:	T Technology > T Technology (General) T Technology > T Technology (General) > T10.5 Communication of technical information
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Engineering > Department of Electrical and Computer Engineering Kulliyyah of Engineering
Depositing User:	Prof. Dr Othman O. Khalifa
Date Deposited:	10 Nov 2026 16:29
Last Update:	10 Nov 2026 16:29
Queue Number:	2026-11-Q6088
URI:	http://irep.iium.edu.my/id/eprint/132544
Indexed In:	Google Scholar

Actions (login required)

View Item