Semi-Automated Framework for Feature Engineering in Machine Learning and Data Analysis

This research introduces the Semi-Automated Feature Engineering framework, an approach to addressing the complexity of feature engineering in machine learning. The framework addresses an important challenge in data science by creating a systematic method that integrates algorithmic techniques with human expertise, aiming to reduce the computational burden and enhance feature generation processes. The framework is designed to provide a structured approach to feature engineering, enabling data scientists to efficiently create and validate features. Its core methodology involves an iterative process that combines automated feature generation with expert-guided selection, allowing for more targeted and meaningful feature creation. To validate the framework's effectiveness, the research conducted experimental trials in medical diagnostics, using the Orinda Longitudinal Study of Myopia and the Diabetic Retinopathy Debrecen Dataset and a genetic algorithm-based approach for feature engineering. Experimental results demonstrated the framework's potential, with generated features showing consistent performance improvements. The research contributes a systematic framework for feature engineering in data analysis, providing data scientists with a method that balances algorithmic efficiency with human expertise.