Gradient Descent Finds the Maximum-Margin Classifier, Slowly: Ten Million Iterations of Unregularised Logistic Regression on Two Separable UCI Data Sets, a Norm–Loss Bound, and Early Stopping

When the training data can be split perfectly by a hyperplane, unregularised logistic regression has no finite solution, yet gradient descent on it is known to converge in direction to the hard-margin support vector machine. The theory says this convergence is logarithmically slow. This paper measures how slow on real data. A linear programme shows that two of four standard UCI data sets are linearly separable after standardisation: breast cancer Wisconsin diagnostic (569 cases, 30 features) and sonar (208, 60); banknote and ionosphere are not. I prove two elementary facts: if the average logistic loss falls below ln 2 divided by n, every training point is correctly classified, and every weight vector obeys a norm–loss bound, so the norm must be at least (1/gamma) ln(ln 2/(n times the loss)) where gamma is the maximum geometric margin. I then run ten million gradient-descent steps on each data set and certify the exact max-margin solution by its KKT conditions. On sonar (margin 0.0196) the iterates separate the data from about one million steps and reach a cosine gap of 0.00226 with the SVM direction. On breast cancer (margin 0.0014) they still misclassify training points after ten million steps, with a cosine gap of 0.060, and the norm grows at 12.5 per unit of log t against the asymptotic 718. The small margin explains the slowness. Out of sample, the slow road is not worth travelling: with five-fold cross-validation, gradient descent stopped at 1,000 steps has error 0.0211 on breast cancer against 0.0475 for the SVM, and stopped at 10 steps, 0.2260 on sonar against 0.2837.