Scalable SubXPCA on a Distributed Platform Applied to Classification of Hyperspectral Images

The exponential growth of technological advancements in satellite and airborne remote sensing is giving rise to large volumes of high-dimensional hyperspectral image data. Apache Spark is one of the most popular, extensively used and open-source distributed processing frameworks, which is proven effective in processing large volumes of remotely sensed hyperspectral images in a time-efficient manner. Open-source distributed processing frameworks have proven effective in processing large volumes of remotely sensed hyperspectral images quickly and efficiently. While computational power has been increasing, the rate of data accumulation is more than the processing capabilities. Therefore, more efficient algorithms such as dimensionality reduction are needed to process and get accurate performance for the application. This paper proposes an efficient and parallel spectral dimensionality reduction approach based on feature partitioning principal component analysis called scalable SubXPCA. We implemented scalable SubXPCA on a spark cluster distributed environment. We compared scalable SubXPCA against other distributed feature partitioning and various non-feature partitioning dimensionality reduction methods. Our experiments on different real and synthetic datasets of hyperspectral images confirm that SubXPCA classification performance is not only better than its competitors but also that the running time of SubXPCA is better in distributed processing than serial processing. As the size of the hyperspectral image dataset increased, SubXPCA showed a speed up factor of 5.7× and more in the spark cluster compared to the serial version.