Naive Bayes or Logistic Regression? Learning Curves on Five UCI Datasets, the Role of Regularisation, and an Exact Analysis of Duplicated Features

A widely cited result of Ng and Jordan says that naive Bayes, a generative classifier, reaches its asymptotic error with fewer training examples than logistic regression, its discriminative counterpart, so naive Bayes should win on small samples and logistic regression on large ones. I test this on five public UCI datasets with learning curves from 10 to 8,000 training examples, 40 random splits each, comparing Gaussian naive Bayes with logistic regression at two penalty strengths. With the default L2 penalty (C=1), logistic regression has the lower mean test error at every training size on four of the five datasets, and naive Bayes is better by more than two standard errors only on the ionosphere data, at 30, 75 and 100 examples. Against weakly penalised logistic regression (C = 10⁴), naive Bayes also wins on ionosphere from 30 to 200 examples and on the MAGIC telescope data at 20 and 30. The small-sample advantage of naive Bayes is therefore mostly an advantage over an unregularised competitor. I also prove three exact results on duplicated features: copying every feature k times multiplies the naive Bayes log-likelihood ratio by exactly k; it leaves unpenalised logistic regression unchanged; and with an L2 penalty it is equivalent to multiplying C by k. The code confirms each to numerical precision. Naive Bayes is also poorly calibrated: its Brier score at the largest training size is between 1.02 and 8.3 times that of logistic regression.