Classification by separating hypersurfaces: An entropic approach

This work introduces a variation on the theme of the classical linear classification problem, extending it to the separation of data by non-linear polynomial hypersurfaces, which allows for more complex decision boundaries. The classification problem, central to machine learning since the perceptron model, is transformed into an ill-posed, linear inverse problem with convex constraints. We solve this using an entropy minimization procedure. Our approach differs from traditional setups as we do not pre-specify the measure of separation for the training data. Instead, the solution itself yields this measure, quantifying the degree of non-separation between the training classes. Crucially, the method intrinsically provides a way to define a region of undecidability (or uncertainty), where points that fall within that region cannot be classified with certainty. This entropic method offers a robust alternative (stable under small data perturbations) to traditional linear or quadratic optimization techniques like Support Vector Machines (SVMs). Furthermore, we explicitly compare our entropic method against classification using Convolutional Neural Networks (CNNs), a prominent class of gradient descent based models. Numerical experiments on diverse datasets, including linear and non-linear cases, demonstrate the efficiency and versatility of the method, plus its competitive performance against both established classical techniques and modern deep learning approaches.