Anomaly detection of denial-of-service network traffic attacks using autoencoders and isolation forest

Ghulam Hussain, Muhammad Thaqif and Shafeeq Lone, Aman and Maspo, Nur-Adib and Attarbashi, Zainab (2026) Anomaly detection of denial-of-service network traffic attacks using autoencoders and isolation forest. International Journal on Perceptive and Cognitive Computing, 12 (1). pp. 145-151. E-ISSN 2462-229X

Preview

PDF - Published Version Download (2MB) | Preview

Abstract

This paper presents an unsupervised network-based anomaly detection framework that integrates deep autoencoders with the Isolation Forest algorithm. The framework analyzes extracted traffic features, including packet length and IP address patterns, to detect deviations from normal behaviour without requiring labelled data. Autoencoders reconstruct benign traffic to highlight subtle deviations, while Isolation Forest efficiently assigns anomaly scores to identify statistical outliers in large-scale, unlabelled datasets. Experimental evaluation shows that the Isolation Forest model achieves a low mean squared error (MSE) of 0.0065 with an accuracy of 9.79%, indicating stable anomaly score separation, whereas the standalone autoencoder records a substantially higher reconstruction error (MSE = 3.92 × 10¹?) and an accuracy of 6.09%, reflecting the difficulty of modelling complex and highly variable network traffic patterns. By combining both approaches, the proposed framework improves overall detection performance, achieving a higher accuracy of 13.55%, and demonstrates enhanced capability in detecting both volumetric and stealthy attacks, such as application-layer denial-of-service (DoS) traffic. Visualization of traffic behaviour further supports the analysis, revealing clearer separation between normal and anomalous flows when both models are integrated. These findings highlight the complementary strengths of statistical outlier detection and deep learning-based reconstruction, providing a practical foundation for adaptive and real-time anomaly monitoring in dynamic network environments

Item Type:	Article (Journal)
Uncontrolled Keywords:	Anomaly Detection, Autoencoder, Isolation Forest, Network Security, Unsupervised Learning
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science Q Science > QA Mathematics > QA76 Computer software T Technology > TK Electrical engineering. Electronics Nuclear engineering > TK7800 Electronics. Computer engineering. Computer hardware. Photoelectronic devices > TK7885 Computer engineering
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Information and Communication Technology Kulliyyah of Information and Communication Technology
Depositing User:	Dr Zainab Senan Mahmod
Date Deposited:	12 Feb 2026 12:09
Last Modified:	12 Feb 2026 12:09
Queue Number:	2026-02-Q2112
URI:	http://irep.iium.edu.my/id/eprint/127399

Actions (login required)

View Item