Controlled Gradient-Flow Ablation in Deep Multilayer Perceptrons

Controlled Gradient-Flow Ablation in Deep Multilayer Perceptrons This study investigates how activation functions, network depth, and weight initialization influence gradient magnitudes across layers in multilayer perceptrons (MLPs). Using a controlled experimental setup implemented in PyTorch, the work examines gradient-flow behavior through three ablation studies. The first study compares Sigmoid, Tanh, and ReLU activation functions in an eight-layer network using Xavier initialization. The second investigates the effect of increasing network depth across 4, 8, 16, and 32 layers using ReLU activation and He initialization. The third compares Xavier and He initialization in an eight-layer network with ReLU activation. Experiments use synthetic inputs and binary targets, with BCEWithLogitsLoss as the loss function. For each experimental configuration, gradients are measured after a single forward and backward pass across three random seeds. The analysis records layer-wise weight-gradient L2 norms and root-mean-square (RMS) magnitudes to characterize differences in gradient scale throughout the network. The study provides a structured diagnostic framework for examining how activation functions, depth, and initialization relate to gradient propagation in deep feed-forward networks. By isolating these factors through controlled ablations, it offers an interpretable way to investigate gradient-scale behavior without confounding the initial measurements with optimizer-driven parameter updates. This work is intended as an exploratory gradient-flow analysis rather than a benchmark of model accuracy or training performance. Since the experiments use synthetic data, a limited number of random seeds, and a single backward pass per run, the findings should be interpreted within the boundaries of the experimental design. Further investigation involving more random seeds, multi-step training, and task-relevant datasets would help establish how these observed gradient patterns relate to practical optimization and learning outcomes. Keywords: Gradient Flow, Vanishing Gradients, Deep Neural Networks, Multilayer Perceptrons, PyTorch, Activation Functions, Weight Initialization, Xavier Initialization, He Initialization, Ablation Study, Backpropagation, Deep Learning Diagnostics. Author: Rajesh PhulwariaAffiliation: Independent Researcher