Real-time Monitoring System for Large-scale Data Processing in Cloud Computing Environment: A Research Study

To meet the real-time processing demands of large-scale e-commerce transaction data, a distributed data processing system was designed and implemented. The system utilizes an improved LZ4 compression algorithm and micro-batch processing mechanism to optimize data transmission efficiency and computing performance. A 25-node cluster environment was set up for performance testing, and the system achieved a single-node processing capacity of 56,700 transactions per second and an overall processing capacity of $\mathbf{8 9, 0 0 0}$ transactions per second. After six months of deployment on a certain e-commerce platform, the system processed an average of 500 million transactions per day, with an $\mathbf{8 5 \%}$ increase in accurate identification of abnormal orders and a system availability of $\mathbf{9 9. 9 9 \%}$, achieving high-efficiency and stable data processing.