Cognitive detection at big-data scale: a CNN-LSTM-DQN framework with prioritized experience replay for cross-attack-family generalization and multi-seed initialization sensitivity analysis

Rustam, Rushendra and Ramli, Kalamullah and Purnamasari, Prima Dewi and Gunawan, Teddy Surya and Salman, Muhammad (2026) Cognitive detection at big-data scale: a CNN-LSTM-DQN framework with prioritized experience replay for cross-attack-family generalization and multi-seed initialization sensitivity analysis. Big Data and Cognitive Computing, 10 (7). p. 239. E-ISSN 2504-2289

	PDF - Published Version Restricted to Repository staff only Download (17MB) | Request a copy
Preview	PDF (WOS) - Supplemental Material Download (650kB) | Preview
Preview	PDF (Scopus) - Supplemental Material Download (205kB) | Preview

Abstract

Real-world IoT network security generates traffic at big-data scale with extreme class imbalance, temporal non-stationarity, and continuously evolving attack strategies that overwhelm static supervised classifiers. This paper presents a cognitive computing framework for network intrusion detection: a CNN–LSTM–DQN architecture with Prioritized Experience Replay (PER) evaluated on a 5,000,000-flow naturalistic sample of the TON_IoT Processed_Network dataset (4,000,000 training/1,000,000 temporally held-out test flows; 94.5% attack ratio) under a strict temporal split. The cognitive agent optimizes detection decisions using an Alerts per Million Flows (ARMF)-aware reward function that encodes both alert-fatigue cost and missed-attack penalty. We conduct a cross-attack-family generalization study: the methodology—architecture template, reward design, and hyperparameter calibration—is inherited from a framework previously validated on CSE-CIC-IDS2018, re-instantiated and retrained on the structurally different TON_IoT environment, and compared against the previously published benchmark. Initialization sensitivity is characterized across five independent random seeds using paired Wilcoxon signed-rank and t-tests. Across the five seeds, the proposed X2 model attains recall 0.833 ± 0.306 and F1 0.874 ± 0.241 (mean ± sample SD), versus the supervised X1 baseline at 0.858 ± 0.178 and 0.912 ± 0.116; the best-performing seed (42) achieves 97.52% accuracy, 98.02% attack recall, 99.46% precision, and 98.73% F1-score on 1,000,000 held-out XSS flows—an attack family entirely absent from training—with temporal stability variances of 4.63 × 10−7 (recall) and 1.38 × 10−7 (F1). The X2 advantage observed among the four stable seeds is not statistically demonstrated at n = 5 (statistical power ≈ 5.1%); the initialization-sensitivity finding itself, including one degenerate alert-suppression seed, is reported as a primary contribution. A formal, exactly additive ARMF decomposition distinguishes the detected-attack (structural) component (99.46%) from the model-induced false-positive component (0.54%), and we report a multi-seed, ARMF-aware cognitive IDS evaluation on naturalistic TON_IoT traffic under an unseen-attack-family test condition that, to the best of our knowledge, has not been reported in the surveyed RL-based NIDS literature.

Item Type:	Article (Journal)
Uncontrolled Keywords:	big data security; cognitive computing; intrusion detection system; deep reinforcement learning; CNN–LSTM; prioritized experience replay; TON_IoT; cross-attack-family generalization; multi-seed sensitivity analysis; ARMF; IoT network security; class imbalance at scale
Subjects:	T Technology > TK Electrical engineering. Electronics Nuclear engineering > TK7800 Electronics. Computer engineering. Computer hardware. Photoelectronic devices > TK7885 Computer engineering
Kulliyyahs/Centres/Divisions/Institutes (Can select more than one option. Press CONTROL button):	Kulliyyah of Engineering > Department of Electrical and Computer Engineering
Depositing User:	Prof. Dr. Teddy Surya Gunawan
Date Deposited:	09 Aug 2026 12:14
Last Update:	09 Aug 2026 12:14
Queue Number:	2026-08-Q4543
URI:	http://irep.iium.edu.my/id/eprint/130561

Actions (login required)

View Item