Abdul Malik, Nurul Emilia Diyana and Mohd. Tamrin, Mohd. Izzuddin and Zainudin, Airul Azizan and Ibrahim, Muhamad Ariff and Che Azemin, Mohd. Zulfaezal (2026) Beyond conventional ASR metrics: evaluating semantic preservation in low-resource speech recognition. International Journal of Advanced Computer Science and Applications, 17 (10). pp. 234-240. ISSN 2158-107X E-ISSN 2156-5570
|
PDF
- Published Version
Download (310kB) | Preview |
Abstract
Word error rate (WER) and character error rate (CER) quantify lexical discrepancies in automatic speech recognition (ASR) but do not directly indicate whether the intended message remains intact. This exploratory study compared lexical accuracy with human-rated preservation of essential meaning in Malay-English code-switched conversational speech. Four complete inference pipelines were evaluated on 30 purposively selected windows from one recording (826 reference words): Mesolitica Malaysian Whisper Large-v3-Turbo-v3 (Pipeline 1), OpenAI Whisper Large-v3-Turbo through fasterwhisper (Pipeline 2), Qwen3-ASR-1.7B with a combined MalayEnglish contextual prompt (Pipeline 3), and the same Qwen3-ASR model without that prompt (Pipeline 4). Three evaluators independently assessed 120 reference-hypothesis pairs; a two-ofthree majority constituted the semantic endpoint. Pipeline 4 had the lowest WER (29.18%) and CER (20.60%), whereas Pipeline 1 had the highest observed majority-rated semantic-preservation proportion (53.3%, versus 36.7% for Pipeline 4). The overall pipeline effect was significant for WER (Friedman chi-square (3) = 15.64, p = 0.0013) and CER (chi-square (3) = 18.17, p < 0.001), but not for semantic preservation (Cochran's Q = 5.57, p = 0.134); no paired semantic contrast remained significant after Holm adjustment. The Pipeline 1–Pipeline 4 semantic difference was 16.7 percentage points (paired-window bootstrap 95% confidence interval: −6.7 to 40.0 percentage points). Overall Fleiss' kappa was 0.529. Different descriptive rankings were therefore observed, but this single-recording sample does not establish significant differences in semantic performance or general model superiority. These findings support further evaluation of human semantic judgments alongside lexical metrics, with independent validation across recordings, speakers, and inference implementations.
| Item Type: | Article (Journal) |
|---|---|
| Uncontrolled Keywords: | Automatic speech recognition; Malay-English codeswitching; low-resource speech; word error rate; character error rate; semantic preservation; human evaluation; exploratory benchmarking |
| Subjects: | Q Science > Q Science (General) |
| Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button): | Kulliyyah of Allied Health Sciences Kulliyyah of Information and Communication Technology Kulliyyah of Information and Communication Technology Kulliyyah of Information and Communication Technology > Department of Information System Kulliyyah of Information and Communication Technology > Department of Information System Kulliyyah of Nursing Kulliyyah of Nursing > Department of Professional Nursing Studies |
| Depositing User: | Dr. Mohd Zulfaezal Che Azemin |
| Date Deposited: | 11 Nov 2026 12:39 |
| Last Update: | 11 Nov 2026 15:59 |
| Queue Number: | 2026-11-Q6089 |
| URI: | http://irep.iium.edu.my/id/eprint/132547 |
| Indexed In: | ERA |
Actions (login required)
 |
View Item |