Real-Time Network Monitoring, Performance and Traffic Analytics Platform: An Intelligent Framework for High-Frequency Telemetry, Statistical Anomaly Detection, and Operational Insights

Micro-bursts and other temporary performance drops cause critical performance bottlenecks for scattered applications and usually emerge long before network monitoring systems can even register any sign of degradation, let alone fix the issue. Classic methods of full packet capture, which are not feasible to use regularly on any network endpoint, and SNMP polling, which cannot be used to track short-lived phenomena, do not help to tackle this problem. Our solution is a lightweight host-based system of telemetry and traffic analysis designed to perform high-frequency network diagnostics. We monitor host OS (using appropriate platform commands for Linux and Windows) to achieve 64-bit I/O readings every half a second and exploit it for calculating utilization of the sending/receiving bandwidth and physical link capabilities. An individual, asynchronous probe forwards non-obstructive ICMP query packets once a second for measuring the round-trip time. Packet loss is determined from the sum of the count of sent packets and received packets. The detection of abnormal events is achieved by utilizing the rolling-Z-score approach and the isolation forest method (unsupervised learning technique), which helps in detecting deviations from a statistical baseline established on the basis of weekly daily data samples. The performance of the proposed solution is assessed using the netem emulation system, showing a high accuracy in identifying declines in the UDP traffic volume, deprivation of TCP traffic, and Latency caused by the NetEm system (with a 97.2% F1-score and 2.1% rate of false alarms).