A Spatiotemporal Masked Pretraining Framework for Nonlinear Dynamic Soft Sensor Development

Data-driven soft sensors are extensively applied to predicting quality indicators. Recent studies have revealed that deep learning models can provide promising means for efficient predictive modeling. Nevertheless, most of them suffer from the intricate characteristics of spatiotemporal coupling relations, which hammers their widespread application in process industries. To handle those characteristics, this paper presents a spatiotemporal masked pre-training (SMPT) framework for soft sensing, which eases the masked modeling and guides the feature representation learning. Specifically, a graph-based memory unit with dynamic graph construction is devised to capture the connections among various process variables, accounting for the temporal and spatial correlations simultaneously. A multiple masked time series-based pre-training strategy is adopted to recover the randomly generated series to the original series via the weighted aggregation of similar neighbors, which assembles the ruined but complementary spatiotemporal information. In addition, a contrastive learning-based network training approach is designed for the effective high-level feature learning and series-wise similarity learning. Finally, SMPT is fine-tuned to obtain an output-feature mapping for predicting quality indicators. Compared with nine state-of-the-art (SOTA) methods, the proposed SMPT empirically presents superior results on two industrial datasets, yielding relative reductions in mean absolute error (MAE) of 18.0% and 9.0% over the second-best methods on the two datasets, respectively.