Linear Regression

This set of slides offers a detailed yet accessible overview of Linear Regression, covering both deterministic and probabilistic approaches. The slides start by introducing the formulation of linear regression, where the output yyy is modeled as a linear combination of basis functions of the input features x\mathbf{x}x. The least squares method is presented as a deterministic approach to minimize the mean squared error (MSE) between predicted and actual values, with a closed-form solution for the optimal parameters. The presentation then shifts to a probabilistic perspective, framing linear regression as a Linear Gaussian Model. The maximum likelihood estimation approach is used to derive the optimal parameters by maximizing the likelihood of observing the given data. This probabilistic view provides a deeper understanding of uncertainty in predictions. The slides also explore Bayesian linear regression, which incorporates prior knowledge about the model parameters. The posterior distribution over the parameters is derived using Bayes' theorem, allowing for a more informative estimate of uncertainty. The maximum a posteriori (MAP) estimate is discussed as a partially Bayesian approach, while the full Bayesian approach provides a predictive distribution by marginalizing over the posterior distribution of the weights. Visual aids, such as diagrams illustrating residual error and the optimization landscape, help clarify key concepts. The slides highlight the advantages of Bayesian methods, such as quantifying uncertainty and incorporating prior knowledge, while also addressing the limitations of deterministic approaches.