Comprehensive Analysis on Feature Selection and Dimensionality Reduction in Big Data

From the past few years, massive volumes of structured, semi-structured and unstructured data are generated across various domains like healthcare, Internet of Things (IoT), finance and multimedia. Removing or ignoring irrelevant attributes from the data reduces the burden on Machine Learning (ML) algorithms. Traditional approaches faced challenges like handling high-dimensional data, scalability issues, limited interpretability, noise, outliers and curse of dimensionality. The need for effective feature selection and dimensionality reduction method improves the model performance by reducing computational costs and enhancing interpretability. In this survey, recent technological approaches based on ML and Deep Learning (DL) are explored for feature selection and dimensionality reduction in big data analytics. The approaches like Principal Component Analysis (PCA), t-distributed Stochastic Neighbour Embedding (t-SNE) and autoencoders efficiently reduce the dimensionality of high-dimensional data from the ImageNet dataset when compared with traditional approaches. The outcome illustrates that ML and DL techniques in this survey are well suited for feature selection and dimensionality reduction in big data analysis, especially in large-scale datasets like ImageNet.