Where Deterministic cuDNN Loses Time on an 8x8 Board: A Bit-Reproducible 1.5x Training Speed-Up for a Compact Recurrent Chess Network on One Pascal GPU

Reproducible training of neural networks usually means running cuDNN in deterministic mode and accepting whatever algorithms its heuristics select. This report examines what that choice costs, and how much of the cost can be recovered without giving up bit-level reproducibility, for one concrete workload: a compact recurrent residual chess network (128 channels, 8 residual blocks applied twice, squeeze-and-excitation and a recurrence gate; 11.05 million parameters) trained by supervised imitation on 8x8 board encodings with batch size 512 on a single GTX 1080 Ti. A kernel-level profile and per-convolution microbenchmarks showed that, in deterministic mode without benchmarking, cuDNN computed the forward pass of the 128-channel 3x3 convolutions with an FFT algorithm taking 1.83-1.92 ms per call, whereas its input-gradient (dX) kernel computed the same sum in 0.75-0.77 ms. Four speed-only options were introduced, each measured alone and then in combination: (i) computing the forward convolution through the dX kernel, using the exact identity between a convolution and a transposed convolution with a channel-swapped, rotated kernel; (ii) fusing the squeeze-and-excitation tail and the recurrence gate with TorchScript; (iii) a hand-written CUDA kernel for the squeeze-and-excitation tail with a fixed-order warp reduction; and (iv) imposing, through a small C++ extension calling cuDNN directly, the fused Winograd dX algorithm, which is deterministic, 20% faster than the algorithm PyTorch selected (0.600 versus 0.754 ms) and closer to a float64 reference. On the full training step the four options reduced the time per step from 173.5 to 111.2 ms (1.56x). In the real training command, the time per step fell from 163.6 ms to 107.6 ms (1.52x), and two identical runs with all four options produced bit-identical checkpoints. Each option passed a prespecified control rule: in 45-minute fine-tuning runs from the same checkpoint and batches, the validation policy cross-entropy at equal step count stayed within 0.003 nat of the control at every checkpoint from 4,000 steps on (worst deviations 0.0019, 0.0018, 0.0023 and 0.0029). Negative results are reported as well. All measurements come from one GPU model, one cuDNN version and one seed per control run; the results are an engineering case study, not a general claim about deterministic training.