Understanding Neural Networks: A Minimalist Implementation from Scratch

This paper explores the foundations of neural networks by implementing a multi-layer perceptron (MLP) from scratch using only NumPy. The primary objective is to gain a deep understanding of the behavior of activation functions (Sigmoid, ReLU) and loss functions (MSE, Binary Cross-Entropy) in the training process. Experiments were conducted on a binary classification dataset for up to 10,000 epochs. Results reveal that while all combinations of activation and loss functions yield comparable accuracy (95-97%), ReLU combined with Binary Cross-Entropy demonstrates the most stable and consistent convergence.