Defining and Benchmarking a Symmetric Kappa+ Prequential Error Metric on Shifting Imbalanced Streaming Data with Label Budgets

Streaming data tasks require that a classifier make label predictions and incrementally re-adapt over the course of the stream. Performance of streaming data classifiers is also temporal, i.e. characterized over a sliding window of the most recent W exemplars. Previous research motivated the comparison of a streaming classifier against a naive ‘no change’ 1-bit classifier (assumes 100% label information) using the Kappa+ Statistic. We generalize the formulation to provide a Statistic that is symmetric, i.e. does not return −∞ as streaming classifier performance degrades. Benchmarking on two challenging streaming classification tasks with multiple non-stationary properties demonstrates the descriptive properties of the resulting Kappa++ Statistic.