Evolutionary Multitasking and Multi-Correlation: Towards More Performant Feature Selection

High-dimensional data in classification problems poses a challenge due to the curse of dimensionality. Feature selection (FS) aims to mitigate this issue by identifying the most relevant features. However, existing FS methods, like PSO, can be computationally expensive and struggle with generalization. This work proposes a solution by introducing a PSO-based FS method enhanced with MFO principles. MFO allows knowledge transfer between tasks, potentially improving generalization. The proposed method generates two related tasks from a single dataset using different correlation measures, promoting knowledge sharing. Additionally, Apache Spark facilitates parallel fitness evaluation, leading to faster computation times. We demonstrate that the proposed method outperforms existing approaches by achieving higher classification accuracy with a smaller feature set while maintaining efficiency.