FCAF: Fisher Score assisted Concrete Autoencoder for Feature Selection

High-dimensional data presents a significant challenge in machine learning, making dimensionality reduction essential. Compared to feature extraction, feature selection preserves the original data structure, reduces information loss, and enhances interpretability. However, feature selection is typically discrete and often requires handcrafted strategies, limiting its use with deep neural networks. Deep neural networks can improve computational efficiency by enabling real-time feature screening and optimization during training, thus saving time and effort in feature engineering. Inspired by the Concrete Autoencoder (CAE), we propose an end-to-end feature selection method based on neural networks. Specifically, we transform the discrete feature selection process into a continuous, differentiable form using the Gumbel-Softmax distribution, allowing the use of backpropagation. Additionally, we incorporate the Fisher score and L2 regularization to select more discriminative categorical features and enhance the stability of model training. Our model takes into account both coarse-grained features at a global level and fine-grained features within individual categories, facilitating a more comprehensive and interpretable feature selection process. Various experiments, including clustering, visualization, and image reconstruction, demonstrate the superiority of the selected features. Furthermore, the application to Few-Shot Learning (FSL) with the selected features illustrates that our model architecture is effective in scenarios with limited data, showcasing its potential for broader applications and providing an interpretable framework for the feature selection process.