Design and Implementation of an Automated Network Traffic Analysis System using Elastic Stack

This paper builds upon our previous work on Network Security Traffic Analysis Platforms (NSTAP) [1], presenting an advanced framework for the real-time monitoring of network traffic and endpoint security in large-scale enterprises. We employ a fully integrated technology stack that includes Elastic Stack, ZEEK, Osquery, Kafka, and GeoLocation data to create a comprehensive security analytics solution. A significant contribution of this research is the integration of supervised machine learning models into our platform, trained specifically on the UNSW-NB15 dataset. We explored three supervised machine learning algorithms - Random Forest (RF), Decision Trees (DT), and Support Vector Machines (SVM). For SVM, we also tested a dimensionality reduction algorithm to maximize model accuracy and optimized both computation time and performance. The evaluation is based on Accuracy, False Positive Rate (FPR) and revealed that the Random Forest Classifier, in conjunction with Pearson correlation-based feature selection methods, achieved the highest accuracy of 99.32% and an error rate of 0.67%.These findings not only substantiate the robustness of our unified platform but also set the stage for future research in developing scalable, efficient, and automated security solutions tailored for large enterprises.