Correlation-Driven Feature Selection and Penalty-Enhanced Autoencoder for Dimensionality Reduction

High-dimensional datasets often pose challenges for data mining and machine learning due to redundant and irrelevant information. In this work, a hybrid strategy is proposed that reduces dimensionality while preserving relevant data and minimizing computation time for tasks like classification. The method combines correlation measures with an auto-encoder, improving data processing, storage, and visualization. Initially, the approach selects the most relevant features using correlation, then applies an auto-encoder to further reduce the dataset while preserving nonlinear relationships. A penalty on the reconstruction error is also proposed to maintain correlation between the input and output layers of the auto-encoder. Experiments demonstrate that the method enhances accuracy and reduces dimensionality without losing information, offering a potential advance in dimensionality reduction techniques.