Distinguishing Order Effects from Random Variation in Cleaning Pipelines for Tabular Data Classification

Abstract Before training a classifier on tabular data, practitioners usually fill in missing values, remove outliers, correct wrong labels and balance classes. Each step changes the data the next one receives, so their order can change accuracy, yet it is usually chosen by convention. Studies that compare orders rank them by average accuracy over a few train--test splits, but with so few splits the best and worst orders differ by chance even when order has no effect. We propose the Order Sensitivity Index (OSI), the relative accuracy gap between the best and worst order, and a permutation test that estimates the gap expected if order did not matter. We evaluated all 24 orders of four steps on ten datasets with three classifiers and known injected errors, and under eight further conditions. In the main experiment, this expected gap exceeded half of every observed gap. Order still mattered beyond chance in 15 of 30 dataset--classifier pairs, changing balanced accuracy by 0.6 to 2.8 percentage points. The effect grew with label noise and was not detected when errors were spread evenly across classes. The most consistent pattern concerned balancing classes before or after label correction. The known errors revealed why: a label-correction model trained on imbalanced data changed many correct minority-class labels. Balancing classes within the correction model's training data prevented much of this, raising balanced accuracy by 1.4 to 1.9 points over correction on imbalanced data. Order comparisons should be tested against chance, and label correction should account for class imbalance.