An Enhanced Machine Learning & Deep Learning based Intrusion Detection System for Intelligent Network Security: A Comprehensive Analysis to Avoid Intrusions in Big Data-based IoT Ecosystem

Abstract

The data growth is measured in exponential rates that are estimated at zettabytes to petabytes globally in the past decade in computer networks and Internet of Things (IoT) networks. The network growth has therefore also caused security issues. Nonetheless, it is difficult to monitor intrusion in this type of big data. Other advanced applications of the emerging networks are smart homes, smart cities, smart grids, smart devices, objects, e-commerce, e-banking, e-government, etc. The security and privacy threats facing most computer networks have led to the development of many Intrusion Detection Systems (IDS) in the recent past. The damage to data confidentiality, integrity, and availability will be experienced in the case of failure of the IDS prevention. The traditional methods are ineffective to match the sophisticated attacks. Rapid advancements in Internet of Things (IoT) infrastructure and Cloud Computing have expanded the digital threat landscape, necessitating a shift from outdated defensive frameworks. First, traditional signature-based systems are unable to identify zero-day attacks. In addition, classical Machine Learning (ML) systems cannot efficiently filter the 5G networks’ encrypted traffic. Competitionally, Deep Learning (DL) systems can automate feature extraction, but individual systems have specific weaknesses. Convolutional Neural Networks (CNN) overlook the importance of temporal dependencies; Recurrent Neural Networks (RNN) are burdened by the vanishing gradient problem, excessive computation costs, and temporal dependencies. Such weaknesses can be alleviated using Hybrid Deep Learning (HDL) systems like CNN-LSTM, CNN-GRU, and Transformers. This paper systematically and critically assesses the recent literature on the “Efficient HDL-Based IDS.” More than just descriptive summaries, we put forth a framework for a taxonomy of Sequential, Parallel, and Auxiliary architectures, which we assess using a Hybrid Efficiency Score (HES). We claim the existence of the “Efficiency-Accuracy Pareto Frontier.” For instance, we position Parallel Ensembles at the bottom, imposing a 63% efficiency cost and Transformer-based and Sequential-Cascading hybrids at the top as real-time ready “Tier 1” systems. We finish the review by providing a reproducibility checklist and a “Green AI” roadmap to support sustainable network security.