ETL Process Efficiency for Predictive Analytics

Purpose of research. This paper investigates the effectiveness of different missing value handling methods in dataframes for data preprocessing tasks in predictive analytics. Three open datasets containing information on building characteristics, meteorological conditions, and energy consumption are used as test data. The goal of the study is to identify the most effective method for data preprocessing in the ETL process for solving predictive analytics problems. Methods. The paper combines dataframes from each dataset and analyzes standard methods of the Pandas module, a high-level library of the Python language, such as direct assignment, the use of indexers, and the fillna method with a dictionary. In addition, a module in Cython, a C-like programming language, is developed to optimize the process of filling missing values, and the performance of each method is evaluated. Results. The results demonstrate that direct assignment is the most effective method in terms of performance in Pandas. Using Cython, although theoretically capable of speeding up calculations, in this case showed a significant decrease in performance due to the overhead of data transformation and interaction between Python and Cython. Code profiling confirmed that the place with insufficient performance is Pandas operations, not Cython code execution. Conclusion. Thus, for most ETL tasks, it is recommended to use optimized Pandas methods, and Cython should be used only in cases of critical need for performance improvement and with careful optimization of the code to minimize overhead, since writing code similar to Pandas will require significant resources, including for its optimization, which in most cases is redundant.