The Spectral‐Spatial Transformer Neural Net for Pixel‐Wise Damage Detection of Agaricus Bisporus Based Hyperspectral Image

ABSTRACT The harvesting and transportation of Agaricus bisporus can lead to surface damage. Non‐destructive detection of mechanical damage in Agaricus bisporus is essential for preserving mushroom freshness, meeting production demands, and minimizing economic losses. It is challenging to accurately detect the damage level of Agaricus bisporus mushrooms in a non‐destructive detection. Therefore, this paper proposes a novel spectral‐spatial transformer (SST) neural network approach. The SST leverages three‐dimensional (3‐D) spectral‐spatial information from hyperspectral images (HSIs) to detect mechanical damage in Agaricus bisporus. The proposed SST method directly extracts spectral‐spatial information and captures spectral local features between adjacent bands through group‐wise spectral embeddings. This grouped information is then fed into the transformer encoder along with positional encoding. Utilizing its multi‐head attention mechanism, the transformer encoder enhances relevant features while suppressing irrelevant ones, thereby improving detection performance. We conducted experiments using support vector machine (SVM), convolutional neural network (CNN), spectral‐spatial feature tokenization transformer (SSFTT), and SST on Agaricus bisporus HSI data. The results demonstrate that SST outperforms SVM and CNN. Specifically, SST achieved an overall accuracy of 98.85% in pixel‐wise classification of Agaricus bisporus. Furthermore, SST accurately identifies both superficial and deep damage, enabling the grading of damaged mushrooms based on the damage ratio, thus fulfilling the requirements of industrial production for Agaricus bisporus. The code of this work is available at https://github.com/Accompagnerzcy/mushroom for the sake of reproducibility.