Where Label Noise Sits: Far-From-Boundary Flips Hurt Gradient Descent More Than Near-Boundary Flips

Most research on label noise in machine learning evaluates how model performance degrades as the overall percentage of corrupted labels increases, while rarely considering where those corrupted labels are located within the feature space. In this paper, I investigate whether the spatial geometry of label noise affects generalization when the total noise rate is held strictly constant. Using synthetic datasets generated from an isotropic Gaussian distribution, I conducted controlled experiments comparing uniform label noise against boundary-concentrated noise across six experimental setups and ten random seeds, keeping the flip count fixed at exactly 20% (k=0.2n). Based on maximum margin theory, I originally hypothesized that corrupting labels near the decision boundary would cause the most harm because gradient descent on logistic loss adjusts decision boundaries relative to points closest to them. However, the results contradicted this hypothesis across every setup tested. Uniform noise, which also places flips far from the true decision boundary, consistently produced higher test error than boundary-concentrated noise. This performance gap expanded substantially as boundary noise was concentrated tighter near the true boundary: for a single-hidden-layer neural network of width 512 at concentration scale 0.25, uniform noise resulted in a +10.03 percentage point higher test error than boundary noise (t=7.33, uniform worse in 10/10 seeds). These findings suggest that under gradient descent, confidently wrong training labels located far from the true boundary exert a far more damaging influence on model generalization than ambiguous label flips near the boundary.