An interpretable real-time outlier detection method for online near infrared tobacco data based on linear regression

In the process of online near infrared (NIR) analysis of tobacco leaves, outliers inevitably appear in the large number of NIR spectra and chemical composition data generated due to factors such as instrumental errors, environmental variations, and sample anomalies. These outliers can severely affect the reliability and validity of subsequent data analysis, and consequently the grading and quality assessment of tobacco leaves. Many existing algorithms are unable to meet the requirements of real-time analysis and interpretability, and some studies considered only anomalies of NIR spectra and neglected the outliers in the predicted chemical composition data. We propose an unsupervised method for online NIR outlier detection based on linear regression using offline data, leveraging the strong positive correlation between nicotine and total nitrogen content in tobacco chemical composition. The method was applied to detect outliers in online tobacco NIR spectra and chemical composition data from different origins in Yunnan Province to evaluate its effectiveness and it was compared with several existing outlier detection algorithms. The results demonstrate that the linear regression method based on offline data can identify and remove outliers in online NIR analysis, thereby significantly improving the reliability and stability of tobacco chemical composition analysis. Compared with several existing outlier detection algorithms, the proposed method exhibits superior real-time performance, lower computational resource consumption, and strong interpretability, making it well-suited for real-time outlier detection in online NIR applications. Notably, in practical application, combining the method with the Mahalanobis distance-based outlier detection can ensure the reliability of the obtained online data, although it will be more stringent.

Authors

Institutions

Publication Details

Journal
Journal of Near Infrared Spectroscopy
Published
2026-11-13
DOI
https://doi.org/10.1177/09670335261505091
Primary Topic
Spectroscopy and Chemometric Analyses
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

An interpretable real-time outlier detection method for online near infrared tobacco data based on linear regression

Hubin Liu, Yadong Wen, Junhui Li, Xiang Ma et al.
2 links
Journal of Near Infrared Spectroscopy
Spectroscopy and Chemometric Analyses
article

An interpretable real-time outlier detection method for online near infrared tobacco data based on linear regression

Hubin Liu, Yadong Wen, Junhui Li, Xiang Ma, Meirong Yin, Luoping Wang
article en

Abstract

In the process of online near infrared (NIR) analysis of tobacco leaves, outliers inevitably appear in the large number of NIR spectra and chemical composition data generated due to factors such as instrumental errors, environmental variations, and sample anomalies. These outliers can severely affect the reliability and validity of subsequent data analysis, and consequently the grading and quality assessment of tobacco leaves. Many existing algorithms are unable to meet the requirements of real-time analysis and interpretability, and some studies considered only anomalies of NIR spectra and neglected the outliers in the predicted chemical composition data. We propose an unsupervised method for online NIR outlier detection based on linear regression using offline data, leveraging the strong positive correlation between nicotine and total nitrogen content in tobacco chemical composition. The method was applied to detect outliers in online tobacco NIR spectra and chemical composition data from different origins in Yunnan Province to evaluate its effectiveness and it was compared with several existing outlier detection algorithms. The results demonstrate that the linear regression method based on offline data can identify and remove outliers in online NIR analysis, thereby significantly improving the reliability and stability of tobacco chemical composition analysis. Compared with several existing outlier detection algorithms, the proposed method exhibits superior real-time performance, lower computational resource consumption, and strong interpretability, making it well-suited for real-time outlier detection in online NIR applications. Notably, in practical application, combining the method with the Mahalanobis distance-based outlier detection can ensure the reliability of the obtained online data, although it will be more stringent.

Journal of Near Infrared Spectroscopy
China Tobacco (CN), China Tobacco Yunnan Industrial Co., Ltd. (China) (CN), China Agricultural University (CN)
Spectroscopy and Chemometric Analyses
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon