Generative AI models: a comparison of application analysis on web AI-based decision support systems for satellite anomaly identification

Mutholib, Abdul and Abdul Rahim, Nadirah and Gunawan, Teddy Surya and Md Yasir, Ahmad Shah Hizam (2026) Generative AI models: a comparison of application analysis on web AI-based decision support systems for satellite anomaly identification. IIUM Engineering Journal, 27 (2). pp. 189-209. ISSN 1511-758X E-ISSN 2289-7860

PDF - Published Version Restricted to Registered users only Download (1MB) | Request a copy

Abstract

The rapid innovation of Generative Artificial Intelligence (GenAI) has transformed Decision Support Systems (DSS) across various domains, including satellite operations. This paper presents a comparative analysis of four free Generative AI models, including Gemma 3 by Google, Llama 4 Maverick by Meta AI, Nemotron Nano 2 by NVIDIA, and Devstral Small by Mistral, in the context of integrating them into a Web AI-based Decision Support System (DSS) for satellite anomaly identification. Using a dataset of over 4,455 satellite anomaly records covering 1957 to 2024 provided by Seradata, with scalability and adaptability to diverse mission profiles. We evaluate these models by generating anomaly analyses for clarity, accuracy, completeness, and relevance across the Incident Overview, Reliability Trend, Insight, and Stakeholder Recommendation categories, using a 5-point Likert scale and Fleiss' Kappa for internal consistency. The comprehensive evaluation of GenAI models for Web AI-based DSS delineates a clear performance stratification, with scores of 4.44 and 4.39 for Nemotron Nano 2 and Llama 4 Maverick, respectively, confirming their positions as the leading systems based on overall Likert scores. However, the analysis further revealed a critical trade-off between absolute quality and internal consistency (Fleiss' Kappa). The superior models, Nemotron Nano 2 and Llama 4 Maverick, achieved high Likert scores by displaying pronounced performance peaks but suffered the lowest internal predictability, with κ = 0.18 on Llama 4 Maverick, indicating a highly volatile output structure in which strong clarity often masked critical incompleteness. Conversely, Devstrall Small, despite its suboptimal mean score of 3.44, demonstrated the highest internal consistency, with κ = 0.66. This robust predictability, even at a lower level, underscores a significant implication for DSS development. The model selection must prioritize the required balance between absolute performance ceiling and the predictability of the output structure. The findings highlight the potential of GenAI implementation in DSS to enhance the reliability of satellite operations, while exploring future directions for research and development in this area. This research contributes to the development of more resilient and intelligent satellite anomaly identification systems, with broader implications for space mission safety, resource optimization, cost reduction, and the future of AI-driven aerospace technologies.

Item Type:	Article (Journal)
Uncontrolled Keywords:	Generative AI, Gemma 3, Llama 4 Maverick, Nemotron Nano 2, Devstral Small, Satellite Anomaly Identification, Clarity, Accuracy, Completeness, Relevance, Decision Support Systems.
Subjects:	T Technology > TK Electrical engineering. Electronics Nuclear engineering T Technology > TK Electrical engineering. Electronics Nuclear engineering > TK5101 Telecommunication. Including telegraphy, radio, radar, television
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Engineering > Department of Electrical and Computer Engineering Kulliyyah of Engineering
Depositing User:	Dr Nadirah Abdul Rahim
Date Deposited:	14 May 2026 10:25
Last Modified:	14 May 2026 10:25
Queue Number:	2026-05-Q3309
URI:	http://irep.iium.edu.my/id/eprint/128940

Actions (login required)

View Item