Softmax-Based Deep Neural Network in Regression

Abstract Regression is a fundamental problem in statistics and machine learning, where the true output is a continuous and stochastic function of the input. Regression methods model the output variables using training datasets composed of input–output pairs derived from real-world scenarios. Inputs are broadly categorized as known or unknown, where unknown inputs are often represented as additive noise. Most existing models rely on two key assumptions: Gaussian residuals and homoscedasticity. While foundational, these assumptions often limit the applicability of traditional machine learning methods. To address these limitations, this study employs deep neural networks (DNNs), leveraging the Universal Approximation Theorem, which ensures their ability to approximate any continuous function given sufficient hidden units. Conventional neural network-based regression methods typically use mean squared error (MSE) or mean absolute error (MAE) as their objective functions, adhering to Gaussian or uniform residual assumptions. However, such methods remain constrained by homoscedasticity and fixed residual probability density functions. Inspired by classification tasks, this study proposes an innovative approach that discretizes continuous regression outputs into bins and models the probability of each bin. The framework utilizes Softmax as the activation function in the output layer and cross-entropy as the objective function, akin to multiclass classification. A comparative analysis against linear regression, Gaussian processes, and conventional MSE-based DNNs is conducted on synthetic datasets with varying complexity and combined cycle power plant (CCPP) dataset. Results demonstrate that the proposed Softmax-based DNN outperforms aforementioned regression methods by effectively estimating conditional probability density functions without having to assume Gaussian or homoscedastic residuals. The Softmax-based DNNs pave the way for a new class of regression models that can more effectively handle the complexities of real-world data.