Classification uncertainty for transient gravitational-wave noise artifacts with optimized conformal prediction

With the increasing use of machine learning (ML) algorithms in scientific research comes the need for reliable uncertainty quantification. When taking a measurement it is not enough to provide the result, we also have to declare how confident we are in the measurement. This is also true when the results are obtained from a ML algorithm, and arguably more so since the internal workings of ML algorithms are often less transparent compared to traditional statistical methods. Additionally, many ML algorithms do not provide uncertainty estimates, and auxiliary algorithms must be applied. Conformal prediction (<a:math xmlns:a="http://www.w3.org/1998/Math/MathML" display="inline"><a:mrow><a:mi>C</a:mi><a:mi>P</a:mi></a:mrow></a:math>) is a framework to provide such uncertainty quantifications for ML point predictors. In this paper, we explore the use and properties of <c:math xmlns:c="http://www.w3.org/1998/Math/MathML" display="inline"><c:mrow><c:mi>C</c:mi><c:mi>P</c:mi></c:mrow></c:math> applied in the context of glitch classification in gravitational wave astronomy. Specifically, we demonstrate the application of <e:math xmlns:e="http://www.w3.org/1998/Math/MathML" display="inline"><e:mrow><e:mi>C</e:mi><e:mi>P</e:mi></e:mrow></e:math> to the Gravity Spy glitch classification algorithm. <g:math xmlns:g="http://www.w3.org/1998/Math/MathML" display="inline"><g:mrow><g:mi>C</g:mi><g:mi>P</g:mi></g:mrow></g:math> makes use of a score function, a nonconformity measure, to convert an algorithm’s heuristic notion of uncertainty to a rigorous uncertainty. We use the application on Gravity Spy to explore the performance of different nonconformity measures and optimize them for our application. Our results show that the optimal nonconformity measure depends on the specific application, as well as the metric used to quantify the performance.