A Novel Model for Soil Organic Matter and Total Nitrogen Detection Based on Visible/Shortwave Near-Infrared Spectroscopy

Soil organic matter (SOM) and total nitrogen (TN) are critical indicators for assessing soil fertility. Although laboratory chemical analysis methods can accurately measure their contents, these techniques are time-consuming and labor-intensive. Spectral technology, characterized by its high sensitivity and convenience, has been increasingly integrated with machine learning algorithms for soil nutrient monitoring. However, the process of spectral data analysis remains complex and requires further optimization for simplicity and efficiency to improve prediction accuracy. This study proposes a novel model to enhance the accuracy of SOM and TN predictions in northeast China’s black soil. Visible/Shortwave Near-Infrared Spectroscopy (Vis/SW-NIRS) data within the 350–1070 nm range were collected, preprocessed, and dimensionality-reduced. The scores of the first nine principal components after a partial least squares (PLS) dimensionality reduction were selected as inputs, and the measured SOM and TN contents were used as outputs to build a back-propagation neural network (BPNN) model. The results show that spectral data processed by the combination of standard normal variate (SNV) and multiple scattering correction (MSC) have the best modeling performance. To improve the accuracy and stability of this model, three algorithms named random search (RS), grid search (GS), and Bayesian optimization (BO) were introduced. The results demonstrate that Vis/SW-NIRS provides reliable predictions of SOM and TN contents, with the PLS-RS-BPNN model achieving the best performance (R2 = 0.980 and 0.972, RMSE = 1.004 and 0.006 for SOM and TN, respectively). Compared to traditional models such as random forests (RF), one-dimensional convolutional neural networks (1D-CNNs), and extreme gradient boosting (XGBoost), the proposed PLS-RS-BPNN model improves R2 by 0.164–0.344 in predicting SOM and by 0.257–0.314 in predicting TN, respectively. These findings confirm the potential of Vis/SW-NIRS technology and the PLS-RS-BPNN model as effective tools for soil nutrient prediction, offering valuable insights for the application of spectral technology in sensing soil information.