An efficient framework for malicious network traffic detection using optimized deep learning techniques

The increasing sophistication of network attacks and the limitations of traditional machine learning-based detection methods pose significant challenges to modern cybersecurity. Existing approaches often depend heavily on labeled data and exhibit poor generalization across heterogeneous environments, limiting their effectiveness against emerging threats. To address these challenges, this paper introduces a novel framework for Malicious Network Traffic Detection (MNTD), designed to improve robustness and adaptability through deep learning methods. The model integrates Convolutional Neural Networks (CNNs), Bidirectional Long Short-Term Memory networks (BiLSTM), and Multi-Head Attention (MHA) mechanisms to capture both spatial and temporal dependencies in traffic. It further employs Adaptive Weighted Delay Velocity (AWDV) for hyperparameter optimization and contrastive learning to enhance feature discrimination, supported by an adaptive loss function and a regularized feature representation strategy to mitigate overfitting. The MNTD framework addresses a binary classification task, distinguishing between benign and malicious traffic. Experimental results demonstrate consistent state-of-the-art performance across four benchmark datasets. On the Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS2017) dataset, it achieves 98. 52% accuracy and a 98. 98% F1-score. On the Canadian Institute for Cybersecurity Domain Name System (DNS) protocol Browser 2020 (CIRA-CIC-DoHBrw2020) dataset, it reaches 98. 82% accuracy and 98. 66% F1-score. For the Botnet Internet of Things (BoT-IoT) dataset, it obtains 98. 65% accuracy and 98. 40% F1-score, while on the University of New South Wales Network Benchmark 2015 (UNSW-NB15) dataset, it maintains 97. 91% accuracy and 97. 61% F1-score. This study demonstrates how artificial intelligence techniques can be effectively applied to cybersecurity applications, specifically malicious network traffic detection.

Authors

• Mukhtar Ahmed
• jinfu chen
• Ernest Akpaku
• Ajmal Latif

Keywords

• Artificial intelligence
• Deep learning
• Malicious traffic detection
• Network security
• Zero-day attack
• Contrastive learning
• Multi-head attention