A multimodal swarm-based multi-agent retrieval-augmented generation (RAG) for domain-specific applications

Ismail, Amelia Ritahani and Afiqa, Nurul and Zulkifli, Muhammad Izzmir Danish and Nur Rashid, Muhammad Ridhwan Irfan (2026) A multimodal swarm-based multi-agent retrieval-augmented generation (RAG) for domain-specific applications. Mathematical Sciences and Informatics Journal, 7 (1). pp. 204-217. E-ISSN 2735-0703

PDF - Published Version Restricted to Repository staff only Download (654kB) | Request a copy

Abstract

The increasing complexity of medical data, encompassing text, images, and documents, requires advanced systems capable of integrating and analyzing multimodal inputs to support clinical decision-making. This study presents a novel multimodal AI system designed for medical information retrieval and decision support, using swarm-based agents and retrieval-augmented generation (RAG) to process diverse data types concurrently. The system employs collective intelligence-inspired swarm agents to distribute tasks efficiently and RAG to enhance contextual relevance. It achieved strong semantic understanding of dental health queries, with BERTScore F1 values of 0.8338 in the query with image upload scenario, though it exhibited challenges in text-image alignment with MM-Align score of 0.3103 and factual consistency of 0.4737. In query-only scenario, dental-related queries have high BERTScore F1 values of 0.9548 and mostly consistent facts of 0.950. While for non-dental queries, the system leveraged web search and give current information as result. The comparative analysis results highlight its adaptability across medical domains, despite limitations in aligning multimodal inputs and ensuring complete factual accuracy. These findings underscore the potential of swarm-based multimodal AI to improve healthcare outcomes by providing contextually relevant analyses while identifying key areas for further refinement

Item Type:	Article (Journal)
Uncontrolled Keywords:	Multimodal AI, Medical information retrieval, Decision support, Swarm-based agents, Retrieval-augmented generation, Dental health
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Information and Communication Technology > Department of Computer Science Kulliyyah of Information and Communication Technology > Department of Computer Science
Depositing User:	Amelia Ritahani Ismail
Date Deposited:	02 Oct 2026 17:07
Last Update:	02 Oct 2026 17:07
Queue Number:	2026-09-Q5389
URI:	http://irep.iium.edu.my/id/eprint/131658
Indexed In:	Google Scholar, MyCite

Actions (login required)

View Item