Ranking Commits by Size Beats Tabular Foundation Models Given Twenty Labels in Just-in-Time Defect Prediction

Just-in-time defect prediction ranks new commits by their risk of introducing a defect, but a new project has few labelled commits to learn from. Tabular foundation models promise good predictions from few labels, so we asked whether they beat a label-free baseline: ranking commits by size, usually lines added. They do not. On 305 open-source projects from seven change-level corpora, ranking by size beat TabPFN v2, TabPFN v3.5 and TabICL, each given 20 labels, by 0.07 to 0.12 on average per corpus in the area under the ROC curve (AUC). On JIT-Defects4J it also beat JITLine, DeepJIT and a CodeBERT-based model trained on the same 20 labels. The foundation models needed at least 100, and often more than 200, labels to catch up. Part of the gap is explained by a short proof: for a model of size alone, the labels only set the direction of the ranking, and when they are few that direction is sometimes wrong. The label generator behind several public benchmarks blames only source-code files, and this makes size counted in code lines look stronger. When we relabelled the commits of 28 newly mined repositories without the file filter, the gain disappeared. For new projects that cannot borrow labels from similar ones, size is the ranking we recommend. Once labels arrive, our size-anchored logistic regression stayed on average within 0.013 AUC of size on JIT-Defects4J and beat the foundation models at every budget, but missed its 0.01 non-inferiority margin at 100 and 200 labels. Preprint, not peer reviewed. Identical to the version submitted to Empirical Software Engineering.