DPCA: Dynamic Probability Calibration Algorithm in LMaaS

Probability calibration is a method to improve the reliability of models by linking the predicted probability to accuracy. Most research follow a static strategy of full fine-tuning. These studies do not consider dynamic data and sparse parameters in Language Models as a Service(LMaaS), leading to limited effectiveness of probability calibration. To address above issues, we propose a dynamic probability calibration algorithm (DPCA) to consider both data flow and parameter freezing. DPCA consists of streaming annotation (SA) task and dynamic calibration (DC) task. The SA task takes a specified number of samples from the training data stream. The sampled data is automatically annotated according to the deviation between predicted probability and true label. The DC task injects the probability deviation of the SA task into next training epoch through adapter-tuning. DPCA achieves data augmentation in LMaaS through joint learning of sample labels and their predicted probability deviation. This work validates DPCA through the implementation on BERT architecutre. The proposed model achieves overall performance improvement on both Chinese and English NLP tasks. Experimental results demonstrate a 2.17% decrease in average ECE without de-creasing in accuracy. Experimental analysis demonstrate the effectiveness and generalizability of DPCA in LMaaS.