A retrieval-augmented generation model for multimodal medical question-answering system

Ismail, Amelia Ritahani and Hishamuddin, Farhan Haikal and Ramjee, Aisar Nasrun and Amir Hussin, Amir 'Aatieff and Afiqa, Nurul (2026) A retrieval-augmented generation model for multimodal medical question-answering system. In: 2025 10th International Conference on Information and Communication Technology for the Muslim World (ICT4M), 26-27 November 2025, KUALA LUMPUR, Malaysia.

PDF - Published Version Restricted to Registered users only Download (1MB) | Request a copy

Abstract

This paper addresses the limitations of existing medical question-answering systems, which are often unimodal and lack retrieval-augmented capabilities or expert-guided learning. To overcome these challenges, a Retrieval-Augmented Generation (RAG) framework was developed to handle multimodal medical data by integrating GPT-2 for text generation and BLIP for visual understanding. The system was fine-tuned using the MedQuAD and VQA-RAD datasets, and a FAISS-based retriever was used to supply relevant external context. Additionally, reinforcement learning from human feedback (RLHF) was applied to align responses with expert knowledge. Experimental results showed that the GPT-2 model achieved a BERTScore F1 of 0.8204, while the multimodal RAG-enhanced GPT-2 model improved to 0.8411, demonstrating the slight effectiveness of combining retrieval and multimodal learning in enhancing medical answer quality. For RAG-enhanced BLIP, the model shows 0.8627 BERTScore with sample question and image.

Item Type:	Proceeding Paper (Plenary Papers)
Uncontrolled Keywords:	Retrieval-Augmented Generation, Multimodal Medical, Question-Answering
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Information and Communication Technology > Department of Computer Science Kulliyyah of Information and Communication Technology > Department of Computer Science Kulliyyah of Information and Communication Technology Kulliyyah of Information and Communication Technology
Depositing User:	Amelia Ritahani Ismail
Date Deposited:	19 Feb 2026 11:50
Last Modified:	19 Feb 2026 11:50
Queue Number:	2026-02-Q2122
URI:	http://irep.iium.edu.my/id/eprint/127404

Actions (login required)

View Item