Automating Data Pipelines in Cloud Infrastructure: The Role of SQL and Python in Real-Time ML Model Deployment

The rapid growth of data and the widespread adoption of cloud computing have driven the need for automated data pipelines to support real-time machine learning (ML) model deployment. This study explores the integration of SQL and Python within cloud infrastructure to build automated, scalable, and efficient data pipelines. By leveraging SQL for structured data management and Python for flexible data processing, the proposed methodology enables seamless data ingestion, transformation, and model deployment in a real-time context. Performance metrics across stages, such as data ingestion, transformation, and model training, indicate significant improvements in throughput, latency, and model accuracy. Statistical validation confirms that optimizations, including query efficiency and memory management, effectively enhance pipeline performance. These findings underscore the value of automated data pipelines in reducing latency and enhancing ML model accuracy, facilitating faster decision-making for real-time applications. This research provides a foundation for future studies on adaptive optimization strategies, privacy considerations, and expanding real-time data sources in automated ML deployments.