Software-Based Compensation of Systematic Errors in a Precision Length Metroscope Using Regression, Interpolation, and Gaussian Process Models

This study investigates software-based compensation of systematic indication errors in a precision Length Metroscope over the measurement range from 1.005 mm to 25 mm. Calibration was performed using 28 reference lengths represented by Grade 1 ceramic gauge blocks, with ten repeated measurements at each calibration point. The initial indication error ranged from 1.49 µm to 4.29 µm, and only 3.6 % of the investigated calibration points had mean indication errors numerically within the nominal ±1.5 µm MPE limits. Five compensation approaches were developed and compared: linear regression, third-order polynomial regression, piecewise linear interpolation, shape-preserving cubic Hermite interpolation (PCHIP), and Gaussian Process Regression (GPR). Predictive performance was evaluated using leave-one-out cross-validation. All methods reduced the original RMSE by more than 80 %. GPR achieved the lowest MAE of 0.287 µm and maximum residual of 0.737 µm, while linear regression provided an almost identical RMSE of 0.379 µm with substantially lower implementation complexity. The results demonstrate that software-based correction can significantly improve the effective metrological performance of existing precision length measuring instruments without mechanical modification.