lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
031d3521814a412ea27ba5a2b63b34d036210a6d
FYS-STK4155/doc/LectureNotes/_build/html/chapter2.html
T
Morten Hjorth-Jensen 031d352181 update
2022-10-05 07:55:07 +02:00

3127 lines
232 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>4. Ridge and Lasso Regression &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet">
<link href="_static/css/index.ff1ffe594081f20da1ef19478df9384b.css" rel="stylesheet">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-book-theme.css?digest=c3fdc42140077d1ad13ad2f1588a4309" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.be7d3bbb2ef33a8344ce.js">
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.d59cb220de22ca1c485ebbdc042f0030.js"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"
const thebe_selector = ".thebe,.cell"
const thebe_selector_input = "pre"
const thebe_selector_output = ".output, .cell_output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="5. Resampling Methods" href="chapter3.html" />
<link rel="prev" title="3. Linear Regression" href="chapter1.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="None">
<!-- Google Analytics -->
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-fluid" id="banner"></div>
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<!-- `logo` is deprecated in Sphinx 4.0, so remove this when we stop supporting 3 -->
<img src="_static/logo.png" class="logo" alt="logo">
<h1 class="site-logo" id="site-title">Applied Data Analysis and Machine Learning</h1>
</a>
</div><form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main">
<div class="bd-toc-item active">
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
About the course
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="schedule.html">
Teaching schedule with links to material
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="teachers.html">
Teachers and Grading
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="textbooks.html">
Textbooks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Review of Statistics with Resampling Techniques and Linear Algebra
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="statistics.html">
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="linalg.html">
2. Linear Algebra, Handling of Arrays and more Python Features
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
From Regression to Support Vector Machines
</span>
</p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
6. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapteroptimization.html">
7. Optimization, the central part of any Machine Learning algortithm
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Decision Trees, Ensemble Methods and Boosting
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Dimensionality Reduction
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="clustering.html">
12. Clustering and Unsupervised Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Deep Learning Methods
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
14. Building a Feed Forward Neural Network
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter11.html">
15. Solving Differential Equations with Deep Learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter12.html">
16. Convolutional Neural Networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter13.html">
17. Recurrent neural networks: Overarching view
</a>
</li>
</ul>
</div>
</nav> <!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="topbar container-xl fixed-top">
<div class="topbar-contents row">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show"></div>
<div class="col pl-md-4 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/chapter2.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.ipynb</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onclick="printPdf(this)" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" aria-label="Fullscreen mode"
title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show noprint">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav" aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#mathematical-interpretation-of-ordinary-least-squares">
4.1. Mathematical Interpretation of Ordinary Least Squares
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-singular-value-decomposition">
4.2. The singular value decomposition
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-math-of-the-svd">
4.3. Basic math of the SVD
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#codes-for-the-svd">
4.4. Codes for the SVD
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#code-for-svd-and-inversion-of-matrices">
4.5. Code for SVD and Inversion of Matrices
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#mathematics-of-the-svd-and-implications">
4.6. Mathematics of the SVD and implications
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#further-properties-important-for-our-analyses-later">
4.7. Further properties (important for our analyses later)
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#meet-the-covariance-matrix">
4.8. Meet the Covariance Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#linking-with-the-svd">
4.9. Linking with the SVD
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id1">
4.10. Ridge and Lasso Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#linking-the-regression-analysis-with-a-statistical-interpretation">
4.11. Linking the regression analysis with a statistical interpretation
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#deriving-ols-from-a-probability-distribution">
4.12. Deriving OLS from a probability distribution
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bayes-theorem-and-ridge-and-lasso-regression">
4.13. Bayes’ Theorem and Ridge and Lasso Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#linking-bayes-theorem-with-ridge-and-lasso-regression">
4.14. Linking Bayes’ Theorem with Ridge and Lasso Regression
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<!-- Table of contents that is only displayed when printing the page -->
<div id="jb-print-docs-body" class="onlyprint">
<h1>Ridge and Lasso Regression</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#mathematical-interpretation-of-ordinary-least-squares">
4.1. Mathematical Interpretation of Ordinary Least Squares
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-singular-value-decomposition">
4.2. The singular value decomposition
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-math-of-the-svd">
4.3. Basic math of the SVD
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#codes-for-the-svd">
4.4. Codes for the SVD
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#code-for-svd-and-inversion-of-matrices">
4.5. Code for SVD and Inversion of Matrices
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#mathematics-of-the-svd-and-implications">
4.6. Mathematics of the SVD and implications
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#further-properties-important-for-our-analyses-later">
4.7. Further properties (important for our analyses later)
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#meet-the-covariance-matrix">
4.8. Meet the Covariance Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#linking-with-the-svd">
4.9. Linking with the SVD
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id1">
4.10. Ridge and Lasso Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#linking-the-regression-analysis-with-a-statistical-interpretation">
4.11. Linking the regression analysis with a statistical interpretation
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#deriving-ols-from-a-probability-distribution">
4.12. Deriving OLS from a probability distribution
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#bayes-theorem-and-ridge-and-lasso-regression">
4.13. Bayes’ Theorem and Ridge and Lasso Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#linking-bayes-theorem-with-ridge-and-lasso-regression">
4.14. Linking Bayes’ Theorem with Ridge and Lasso Regression
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div>
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html chapter2.do.txt --><div class="tex2jax_ignore mathjax_ignore section" id="ridge-and-lasso-regression">
<h1><span class="section-number">4. </span>Ridge and Lasso Regression<a class="headerlink" href="#ridge-and-lasso-regression" title="Permalink to this headline"></a></h1>
<div class="section" id="mathematical-interpretation-of-ordinary-least-squares">
<h2><span class="section-number">4.1. </span>Mathematical Interpretation of Ordinary Least Squares<a class="headerlink" href="#mathematical-interpretation-of-ordinary-least-squares" title="Permalink to this headline"></a></h2>
<p>What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).</p>
<p>We have shown that in ordinary least squares (OLS) the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> are given by</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>The <strong>hat</strong> over <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> means we have the optimal parameters after minimization of the cost function.</p>
<p>This means that our best model is defined as</p>
<div class="math notranslate nohighlight">
\[
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>We now define a matrix</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T.
\]</div>
<p>We can rewrite</p>
<div class="math notranslate nohighlight">
\[
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
\]</div>
<p>The matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> has the important property that <span class="math notranslate nohighlight">\(\boldsymbol{A}^2=\boldsymbol{A}\)</span>. This is the definition of a <a class="reference external" href="https://en.wikipedia.org/wiki/Projection_matrix">projection matrix</a>.
We can then interpret our optimal model <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> as being represented by an orthogonal projection of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> onto a space defined by the column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. In our case here the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.</p>
<p>We have defined the residual error as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}.
\]</div>
<p>The residual errors are then the projections of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> onto the orthogonal component of the space defined by the column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>If the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is an orthogonal (or unitary in case of complex values) matrix, we have</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{X}\boldsymbol{X}^T = \boldsymbol{I}.
\]</div>
<p>In this case the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> becomes</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I},
\]</div>
<p>and we have the obvious case</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0.
\]</div>
<p>This serves also as a useful test of our codes.</p>
</div>
<div class="section" id="the-singular-value-decomposition">
<h2><span class="section-number">4.2. </span>The singular value decomposition<a class="headerlink" href="#the-singular-value-decomposition" title="Permalink to this headline"></a></h2>
<p>The examples we have looked at so far are cases where we normally can
invert the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>. Using a polynomial expansion where we fit of various functions leads to
row vectors of the design matrix which are essentially orthogonal due
to the polynomial character of our model. Obtaining the inverse of the
design matrix is then often done via a so-called LU, QR or Cholesky
decomposition.</p>
<p>As we will also see in the first project,
this may
however not the be case in general and a standard matrix inversion
algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.</p>
<p>There is however a way to circumvent this problem and also
gain some insights about the ordinary least squares approach, and
later shrinkage methods like Ridge and Lasso regressions.</p>
<p>This is given by the <strong>Singular Value Decomposition</strong> (SVD) algorithm,
perhaps the most powerful linear algebra algorithm. The SVD provides
a numerically stable matrix decomposition that is used in a large
swath of applications and the decomposition is always stable
numerically.</p>
<p>In machine learning it plays a central role in dealing with for
example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related
to the covariance matrix (and thereby the correlation matrix) and in
turn the variance of a given quantity. It plays also an important role
in the principal component analysis where high-dimensional data can be
reduced to the statistically relevant features.</p>
<p>One of the typical problems we encounter with linear regression, in particular
when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> (our so-called design matrix) is high-dimensional,
are problems with near singular or singular matrices. The column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>
may be linearly dependent, normally referred to as super-collinearity.<br />
This means that the matrix may be rank deficient and it is basically impossible to
model the data using linear regression. As an example, consider the matrix</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
\mathbf{X} &amp; = \left[
\begin{array}{rrr}
1 &amp; -1 &amp; 2
\\
1 &amp; 0 &amp; 1
\\
1 &amp; 2 &amp; -1
\\
1 &amp; 1 &amp; 0
\end{array} \right]
\end{align*}
\end{split}\]</div>
<p>The columns of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> are linearly dependent. We see this easily since the
the first column is the row-wise sum of the other two columns. The rank (more correct,
the column rank) of a matrix is the dimension of the space spanned by the
column vectors. Hence, the rank of <span class="math notranslate nohighlight">\(\mathbf{X}\)</span> is equal to the number
of linearly independent columns. In this particular case the matrix has rank 1.</p>
<p>Super-collinearity of an <span class="math notranslate nohighlight">\((n \times p)\)</span>-dimensional design matrix <span class="math notranslate nohighlight">\(\mathbf{X}\)</span> implies
that the inverse of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
\boldsymbol{X} &amp; = \left[
\begin{array}{rr}
1 &amp; -1
\\
1 &amp; -1
\end{array} \right].
\end{align*}
\end{split}\]</div>
<p>We see easily that <span class="math notranslate nohighlight">\(\mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0\)</span>. Hence, <span class="math notranslate nohighlight">\(\mathbf{X}\)</span> is singular and its inverse is undefined.
This is equivalent to saying that the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> has at least an eigenvalue which is zero.</p>
<p>If our design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which enters the linear regression problem</p>
<!-- Equation labels as ordinary links -->
<div id="_auto1"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}
\boldsymbol{\beta} = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
\label{_auto1} \tag{1}
\end{equation}
\]</div>
<p>has linearly dependent column vectors, we will not be able to compute the inverse
of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> and we cannot find the parameters (estimators) <span class="math notranslate nohighlight">\(\beta_i\)</span>.
The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})\)</span> can be inverted.
This is more likely to happen when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is high-dimensional. In this case it is likely to encounter a situation where
the regression parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> cannot be estimated.</p>
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I},
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{I}\)</span> is the identity matrix. When we discuss <strong>Ridge</strong> regression this is actually what we end up evaluating. The parameter <span class="math notranslate nohighlight">\(\lambda\)</span> is called a hyperparameter. More about this later.</p>
</div>
<div class="section" id="basic-math-of-the-svd">
<h2><span class="section-number">4.3. </span>Basic math of the SVD<a class="headerlink" href="#basic-math-of-the-svd" title="Permalink to this headline"></a></h2>
<p>From standard linear algebra we know that a square matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> can be diagonalized if and only if it is
a so-called <a class="reference external" href="https://en.wikipedia.org/wiki/Normal_matrix">normal matrix</a>, that is if <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times n}\)</span>
we have <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X}\)</span> or if <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{C}}^{n\times n}\)</span> we have <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X}\)</span>.
The matrix has then a set of eigenpairs</p>
<div class="math notranslate nohighlight">
\[
(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n),
\]</div>
<p>and the eigenvalues are given by the diagonal matrix</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n).
\]</div>
<p>The matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> can be written in terms of an orthogonal/unitary transformation <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span></p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
\]</div>
<p>with <span class="math notranslate nohighlight">\(\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I}\)</span> or <span class="math notranslate nohighlight">\(\boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I}\)</span>.</p>
<p>Not all square matrices are diagonalizable. A matrix like the one discussed above</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X} = \begin{bmatrix}
1&amp; -1 \\
1&amp; -1\\
\end{bmatrix}
\end{split}\]</div>
<p>is not diagonalizable, it is a so-called <a class="reference external" href="https://en.wikipedia.org/wiki/Defective_matrix">defective matrix</a>. It is easy to see that the condition
<span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X}\)</span> is not fulfilled.</p>
<p>However, and this is the strength of the SVD algorithm, any general
matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> can be decomposed in terms of a diagonal matrix and
two orthogonal/unitary matrices. The <a class="reference external" href="https://en.wikipedia.org/wiki/Singular_value_decomposition">Singular Value Decompostion
(SVD) theorem</a>
states that a general <span class="math notranslate nohighlight">\(m\times n\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> can be written in
terms of a diagonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> of dimensionality <span class="math notranslate nohighlight">\(m\times n\)</span>
and two orthognal matrices <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>, where the first has
dimensionality <span class="math notranslate nohighlight">\(m \times m\)</span> and the last dimensionality <span class="math notranslate nohighlight">\(n\times n\)</span>.
We have then</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T
\]</div>
<p>As an example, the above defective matrix can be decomposed as</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1&amp; 1 \\ 1&amp; -1\\ \end{bmatrix} \begin{bmatrix} 2&amp; 0 \\ 0&amp; 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1&amp; -1 \\ 1&amp; 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
\end{split}\]</div>
<p>with eigenvalues <span class="math notranslate nohighlight">\(\sigma_1=2\)</span> and <span class="math notranslate nohighlight">\(\sigma_2=0\)</span>.
The SVD exits always!</p>
<p>The SVD
decomposition (singular values) gives eigenvalues
<span class="math notranslate nohighlight">\(\sigma_i\geq\sigma_{i+1}\)</span> for all <span class="math notranslate nohighlight">\(i\)</span> and for dimensions larger than <span class="math notranslate nohighlight">\(i=p\)</span>, the
eigenvalues (singular values) are zero.</p>
<p>In the general case, where our design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> has dimension
<span class="math notranslate nohighlight">\(n\times p\)</span>, the matrix is thus decomposed into an <span class="math notranslate nohighlight">\(n\times n\)</span>
orthogonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span>, a <span class="math notranslate nohighlight">\(p\times p\)</span> orthogonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>
and a diagonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> with <span class="math notranslate nohighlight">\(r=\mathrm{min}(n,p)\)</span>
singular values <span class="math notranslate nohighlight">\(\sigma_i\geq 0\)</span> on the main diagonal and zeros filling
the rest of the matrix. There are at most <span class="math notranslate nohighlight">\(p\)</span> singular values
assuming that <span class="math notranslate nohighlight">\(n &gt; p\)</span>. In our regression examples for the nuclear
masses and the equation of state this is indeed the case, while for
the Ising model we have <span class="math notranslate nohighlight">\(p &gt; n\)</span>. These are often cases that lead to
near singular or singular matrices.</p>
<p>The columns of <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> are called the left singular vectors while the columns of <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> are the right singular vectors.</p>
<p>If we assume that <span class="math notranslate nohighlight">\(n &gt; p\)</span>, then our matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> has dimension <span class="math notranslate nohighlight">\(n
\times n\)</span>. The last <span class="math notranslate nohighlight">\(n-p\)</span> columns of <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> become however
irrelevant in our calculations since they are multiplied with the
zeros in <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span>.</p>
<p>The economy-size decomposition removes extra rows or columns of zeros
from the diagonal matrix of singular values, <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span>, along with the columns
in either <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> or <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> that multiply those zeros in the expression.
Removing these zeros and columns can improve execution time
and reduce storage requirements without compromising the accuracy of
the decomposition.</p>
<p>If <span class="math notranslate nohighlight">\(n &gt; p\)</span>, we keep only the first <span class="math notranslate nohighlight">\(p\)</span> columns of <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> has dimension <span class="math notranslate nohighlight">\(p\times p\)</span>.
If <span class="math notranslate nohighlight">\(p &gt; n\)</span>, then only the first <span class="math notranslate nohighlight">\(n\)</span> columns of <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> are computed and <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> has dimension <span class="math notranslate nohighlight">\(n\times n\)</span>.
The <span class="math notranslate nohighlight">\(n=p\)</span> case is obvious, we retain the full SVD.
In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy.</p>
</div>
<div class="section" id="codes-for-the-svd">
<h2><span class="section-number">4.4. </span>Codes for the SVD<a class="headerlink" href="#codes-for-the-svd" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="c1"># SVD inversion</span>
<span class="k">def</span> <span class="nf">SVD</span><span class="p">(</span><span class="n">A</span><span class="p">):</span>
<span class="sd">&#39;&#39;&#39; Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).</span>
<span class="sd"> SVD is numerically more stable than the inversion algorithms provided by</span>
<span class="sd"> numpy and scipy.linalg at the cost of being slower.</span>
<span class="sd"> &#39;&#39;&#39;</span>
<span class="n">U</span><span class="p">,</span> <span class="n">S</span><span class="p">,</span> <span class="n">VT</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">svd</span><span class="p">(</span><span class="n">A</span><span class="p">,</span><span class="n">full_matrices</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;test U&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">U</span><span class="p">)</span> <span class="o">@</span> <span class="n">U</span> <span class="o">-</span> <span class="n">U</span> <span class="nd">@np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">U</span><span class="p">)))</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;test VT&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">VT</span><span class="p">)</span> <span class="o">@</span> <span class="n">VT</span> <span class="o">-</span> <span class="n">VT</span> <span class="nd">@np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">VT</span><span class="p">)))</span>
<span class="nb">print</span><span class="p">(</span><span class="n">U</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">S</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">VT</span><span class="p">)</span>
<span class="n">D</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">U</span><span class="p">),</span><span class="nb">len</span><span class="p">(</span><span class="n">VT</span><span class="p">)))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="nb">len</span><span class="p">(</span><span class="n">VT</span><span class="p">)):</span>
<span class="n">D</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">i</span><span class="p">]</span><span class="o">=</span><span class="n">S</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="k">return</span> <span class="n">U</span> <span class="o">@</span> <span class="n">D</span> <span class="o">@</span> <span class="n">VT</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span><span class="o">-</span><span class="mf">1.0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span><span class="o">-</span><span class="mf">1.0</span><span class="p">]])</span>
<span class="c1">#X = np.array([[1, 2], [3, 4], [5, 6]])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="n">C</span> <span class="o">=</span> <span class="n">SVD</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="c1"># Print the difference between the original matrix and the SVD one</span>
<span class="nb">print</span><span class="p">(</span><span class="n">C</span><span class="o">-</span><span class="n">X</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 1. -1.]
[ 1. -1.]]
test U
[[0. 0.]
[0. 0.]]
test VT
[[0. 0.]
[0. 0.]]
[[-0.70710678 -0.70710678]
[-0.70710678 0.70710678]]
[2.00000000e+00 3.35470445e-17]
[[-0.70710678 0.70710678]
[ 0.70710678 0.70710678]]
[[-3.33066907e-16 4.44089210e-16]
[ 0.00000000e+00 2.22044605e-16]]
</pre></div>
</div>
</div>
</div>
<p>The matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> has columns that are linearly dependent. The first
column is the row-wise sum of the other two columns. The rank of a
matrix (the column rank) is the dimension of space spanned by the
column vectors. The rank of the matrix is the number of linearly
independent columns, in this case just <span class="math notranslate nohighlight">\(2\)</span>. We see this from the
singular values when running the above code. Running the standard
inversion algorithm for matrix inversion with <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> results
in the program terminating due to a singular matrix.</p>
<p>The <span class="math notranslate nohighlight">\(U\)</span>, <span class="math notranslate nohighlight">\(S\)</span>, and <span class="math notranslate nohighlight">\(V\)</span> matrices returned from the <strong>svd()</strong> function
cannot be multiplied directly.</p>
<p>As you can see from the code, the <span class="math notranslate nohighlight">\(S\)</span> vector must be converted into a
diagonal matrix. This may cause a problem as the size of the matrices
do not fit the rules of matrix multiplication, where the number of
columns in a matrix must match the number of rows in the subsequent
matrix.</p>
<p>If you wish to include the zero singular values, you will need to
resize the matrices and set up a diagonal matrix as done in the above
example</p>
</div>
<div class="section" id="code-for-svd-and-inversion-of-matrices">
<h2><span class="section-number">4.5. </span>Code for SVD and Inversion of Matrices<a class="headerlink" href="#code-for-svd-and-inversion-of-matrices" title="Permalink to this headline"></a></h2>
<p>How do we use the SVD to invert a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> which is singular or near singular?
The simple answer is to use the linear algebra function for the pseudoinverse, that is</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1">#Ainv = np.linlag.pinv(A)</span>
</pre></div>
</div>
</div>
</div>
<p>Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="c1"># SVD inversion</span>
<span class="k">def</span> <span class="nf">SVDinv</span><span class="p">(</span><span class="n">A</span><span class="p">):</span>
<span class="sd">&#39;&#39;&#39; Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).</span>
<span class="sd"> SVD is numerically more stable than the inversion algorithms provided by</span>
<span class="sd"> numpy and scipy.linalg at the cost of being slower.</span>
<span class="sd"> &#39;&#39;&#39;</span>
<span class="n">U</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">VT</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">svd</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;test U&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">U</span><span class="p">)</span> <span class="o">@</span> <span class="n">U</span> <span class="o">-</span> <span class="n">U</span> <span class="nd">@np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">U</span><span class="p">)))</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;test VT&#39;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">VT</span><span class="p">)</span> <span class="o">@</span> <span class="n">VT</span> <span class="o">-</span> <span class="n">VT</span> <span class="nd">@np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">VT</span><span class="p">)))</span>
<span class="n">D</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">U</span><span class="p">),</span><span class="nb">len</span><span class="p">(</span><span class="n">VT</span><span class="p">)))</span>
<span class="n">D</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">diag</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="n">UT</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">U</span><span class="p">);</span> <span class="n">V</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">VT</span><span class="p">);</span> <span class="n">invD</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">D</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">V</span><span class="p">,</span><span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">invD</span><span class="p">,</span><span class="n">UT</span><span class="p">))</span>
<span class="c1"># Non-singular square matrix</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span> <span class="p">[</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">],[</span><span class="mi">2</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">],[</span><span class="mi">3</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">]])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="n">A</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">X</span><span class="p">)</span> <span class="o">@</span> <span class="n">X</span>
<span class="c1"># Brute force inversion</span>
<span class="n">B</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">A</span><span class="p">)</span> <span class="c1"># here we could use np.linalg.inv(A), try it!</span>
<span class="n">C</span> <span class="o">=</span> <span class="n">SVDinv</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">B</span><span class="o">-</span><span class="n">C</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[1 2 3]
[2 4 5]
[3 5 6]]
test U
[[-2.22044605e-16 -7.49932427e-16 -8.23408962e-16]
[-7.49932427e-16 0.00000000e+00 4.77954956e-17]
[-8.23408962e-16 4.77954956e-17 2.22044605e-16]]
test VT
[[ 3.33066907e-16 -7.32066545e-17 3.32714903e-16]
[-7.32066545e-17 0.00000000e+00 -1.82997013e-16]
[ 3.32714903e-16 -1.82997013e-16 -3.33066907e-16]]
[[0. 0. 0.]
[0. 0. 0.]
[0. 0. 0.]]
</pre></div>
</div>
</div>
</div>
<p>Although our matrix to invert <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> is a square matrix, our matrix may be singular.</p>
<p>The pseudoinverse is the generalization of the matrix inverse for square matrices to
rectangular matrices where the number of rows and columns are not equal.</p>
<p>It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.</p>
<p>Using the SVD we can obtain the pseudoinverse (PI) of a matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> (labeled here as <span class="math notranslate nohighlight">\(\boldsymbol{A}_{\mathrm{PI}}\)</span></p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{D}_{\mathrm{PI}}\)</span> can be calculated by creating a diagonal matrix from <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="c1"># SVD inversion</span>
<span class="k">def</span> <span class="nf">SVDinv</span><span class="p">(</span><span class="n">A</span><span class="p">):</span>
<span class="n">U</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">VT</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">svd</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
<span class="c1"># reciprocals of singular values of s</span>
<span class="n">d</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="n">s</span>
<span class="c1"># create m x n D matrix</span>
<span class="n">D</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">A</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="c1"># populate D with n x n diagonal matrix</span>
<span class="n">D</span><span class="p">[:</span><span class="n">A</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="p">:</span><span class="n">A</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">diag</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
<span class="n">UT</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">U</span><span class="p">)</span>
<span class="n">V</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">VT</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">V</span><span class="p">,</span><span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">D</span><span class="o">.</span><span class="n">T</span><span class="p">,</span><span class="n">UT</span><span class="p">))</span>
<span class="n">A</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span> <span class="p">[</span><span class="mf">0.3</span><span class="p">,</span> <span class="mf">0.4</span><span class="p">],</span> <span class="p">[</span><span class="mf">0.5</span><span class="p">,</span> <span class="mf">0.6</span><span class="p">],</span> <span class="p">[</span><span class="mf">0.7</span><span class="p">,</span> <span class="mf">0.8</span><span class="p">],[</span><span class="mf">0.9</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">]])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
<span class="c1"># Brute force inversion of super-collinear matrix</span>
<span class="n">B</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">B</span><span class="p">)</span>
<span class="c1"># Compare our own algorithm with pinv</span>
<span class="n">C</span> <span class="o">=</span> <span class="n">SVDinv</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">abs</span><span class="p">(</span><span class="n">C</span><span class="o">-</span><span class="n">B</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[0.3 0.4]
[0.5 0.6]
[0.7 0.8]
[0.9 1. ]]
[[-13. -6. 1. 8. ]
[ 11.5 5.5 -0.5 -6.5]]
[[0. 0. 0. 0.]
[0. 0. 0. 0.]]
</pre></div>
</div>
</div>
</div>
<p>As you can see from these examples, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by <strong>Numpy</strong>.</p>
</div>
<div class="section" id="mathematics-of-the-svd-and-implications">
<h2><span class="section-number">4.6. </span>Mathematics of the SVD and implications<a class="headerlink" href="#mathematics-of-the-svd-and-implications" title="Permalink to this headline"></a></h2>
<p>Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.</p>
<p>Our starting point is our design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> of dimension <span class="math notranslate nohighlight">\(n\times p\)</span></p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X}=\begin{bmatrix}
x_{0,0} &amp; x_{0,1} &amp; x_{0,2}&amp; \dots &amp; \dots x_{0,p-1}\\
x_{1,0} &amp; x_{1,1} &amp; x_{1,2}&amp; \dots &amp; \dots x_{1,p-1}\\
x_{2,0} &amp; x_{2,1} &amp; x_{2,2}&amp; \dots &amp; \dots x_{2,p-1}\\
\dots &amp; \dots &amp; \dots &amp; \dots \dots &amp; \dots \\
x_{n-2,0} &amp; x_{n-2,1} &amp; x_{n-2,2}&amp; \dots &amp; \dots x_{n-2,p-1}\\
x_{n-1,0} &amp; x_{n-1,1} &amp; x_{n-1,2}&amp; \dots &amp; \dots x_{n-1,p-1}\\
\end{bmatrix}.
\end{split}\]</div>
<p>We can SVD decompose our matrix as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> is an orthogonal matrix of dimension <span class="math notranslate nohighlight">\(n\times n\)</span>, meaning that <span class="math notranslate nohighlight">\(\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n\)</span>. Here <span class="math notranslate nohighlight">\(\boldsymbol{I}_n\)</span> is the unit matrix of dimension <span class="math notranslate nohighlight">\(n \times n\)</span>.</p>
<p>Similarly, <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> is an orthogonal matrix of dimension <span class="math notranslate nohighlight">\(p\times p\)</span>, meaning that <span class="math notranslate nohighlight">\(\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p\)</span>. Here <span class="math notranslate nohighlight">\(\boldsymbol{I}_p\)</span> is the unit matrix of dimension <span class="math notranslate nohighlight">\(p \times p\)</span>.</p>
<p>Finally <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> contains the singular values <span class="math notranslate nohighlight">\(\sigma_i\)</span>. This matrix has dimension <span class="math notranslate nohighlight">\(n\times p\)</span> and the singular values <span class="math notranslate nohighlight">\(\sigma_i\)</span> are all positive. The non-zero values are ordered in descending order, that is</p>
<div class="math notranslate nohighlight">
\[
\sigma_0 &gt; \sigma_1 &gt; \sigma_2 &gt; \dots &gt; \sigma_{p-1} &gt; 0.
\]</div>
<p>All values beyond <span class="math notranslate nohighlight">\(p-1\)</span> are all zero.</p>
<p>As an example, consider the following <span class="math notranslate nohighlight">\(3\times 2\)</span> example for the matrix <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span></p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{\Sigma}=
\begin{bmatrix}
2&amp; 0 \\
0 &amp; 1 \\
0 &amp; 0 \\
\end{bmatrix}
\end{split}\]</div>
<p>The singular values are <span class="math notranslate nohighlight">\(\sigma_0=2\)</span> and <span class="math notranslate nohighlight">\(\sigma_1=1\)</span>. It is common to rewrite the matrix <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> as</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{\Sigma}=
\begin{bmatrix}
\boldsymbol{\tilde{\Sigma}}\\
\boldsymbol{0}\\
\end{bmatrix},
\end{split}\]</div>
<p>where</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{\tilde{\Sigma}}=
\begin{bmatrix}
2&amp; 0 \\
0 &amp; 1 \\
\end{bmatrix},
\end{split}\]</div>
<p>contains only the singular values. Note also (and we will use this below) that</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=
\begin{bmatrix}
4&amp; 0 \\
0 &amp; 1 \\
\end{bmatrix},
\end{split}\]</div>
<p>which is a <span class="math notranslate nohighlight">\(2\times 2 \)</span> matrix while</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T=
\begin{bmatrix}
4&amp; 0 &amp; 0\\
0 &amp; 1 &amp; 0\\
0 &amp; 0 &amp; 0\\
\end{bmatrix},
\end{split}\]</div>
<p>is a <span class="math notranslate nohighlight">\(3\times 3 \)</span> matrix. The last row and column of this last matrix
contain only zeros. This will have important consequences for our SVD
decomposition of the design matrix.</p>
<p>The matrix that may cause problems for us is <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>. Using the SVD we can rewrite this matrix as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
\]</div>
<p>and using the orthogonality of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
\]</div>
<p>We define <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2\)</span> which is a diagonal matrix containing only the singular values squared. It has dimensionality <span class="math notranslate nohighlight">\(p \times p\)</span>.</p>
<p>We can now insert the result for the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> into our equation for ordinary least squares where</p>
<div class="math notranslate nohighlight">
\[
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\]</div>
<p>and using our SVD decomposition of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\left(\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^{2}(\boldsymbol{V}^T\right)^{-1}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y},
\]</div>
<p>which gives us, using the orthogonality of the matrices <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>,,</p>
<div class="math notranslate nohighlight">
\[
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y},
\]</div>
<p>It means that the ordinary least square model (with the optimal parameters) <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span>, corresponds to an orthogonal transformation of the output (or target) vector <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> by the vectors of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span>.</p>
</div>
<div class="section" id="further-properties-important-for-our-analyses-later">
<h2><span class="section-number">4.7. </span>Further properties (important for our analyses later)<a class="headerlink" href="#further-properties-important-for-our-analyses-later" title="Permalink to this headline"></a></h2>
<p>Let us study again <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> in terms of our SVD,</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
\]</div>
<p>If we now multiply from the right with <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> (using the orthogonality of <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>) we get</p>
<div class="math notranslate nohighlight">
\[
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}.
\]</div>
<p>This means the vectors <span class="math notranslate nohighlight">\(\boldsymbol{v}_i\)</span> of the orthogonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> are the eigenvectors of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>
with eigenvalues given by the singular values squared, that is</p>
<div class="math notranslate nohighlight">
\[
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
\]</div>
<p>Similarly, if we use the SVD decomposition for the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T.
\]</div>
<p>If we now multiply from the right with <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> (using the orthogonality of <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span>) we get</p>
<div class="math notranslate nohighlight">
\[
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T.
\]</div>
<p>This means the vectors <span class="math notranslate nohighlight">\(\boldsymbol{u}_i\)</span> of the orthogonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> are the eigenvectors of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T\)</span>
with eigenvalues given by the singular values squared, that is</p>
<div class="math notranslate nohighlight">
\[
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2.
\]</div>
<p><strong>Important note</strong>: we have defined our design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> to be an
<span class="math notranslate nohighlight">\(n\times p\)</span> matrix. In most supervised learning cases we have that <span class="math notranslate nohighlight">\(n
\ge p\)</span>, and quite often we have <span class="math notranslate nohighlight">\(n &gt;&gt; p\)</span>. For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations).</p>
<p>In our lectures, the number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix and its relation to the SVD.</p>
</div>
<div class="section" id="meet-the-covariance-matrix">
<h2><span class="section-number">4.8. </span>Meet the Covariance Matrix<a class="headerlink" href="#meet-the-covariance-matrix" title="Permalink to this headline"></a></h2>
<p>Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.</p>
<p>We have already noted that the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> in ordinary
least squares is proportional to the second derivative of the cost
function, that is we have</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\]</div>
<p>This quantity defines what is called the Hessian matrix (the second derivative of the cost function we want to optimize).</p>
<p>The Hessian matrix plays an important role and is defined in this course as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
\]</div>
<p>The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.</p>
<p>Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about
the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods.</p>
<p>Suppose we have defined two vectors
<span class="math notranslate nohighlight">\(\hat{x}\)</span> and <span class="math notranslate nohighlight">\(\hat{y}\)</span> with <span class="math notranslate nohighlight">\(n\)</span> elements each. The covariance matrix <span class="math notranslate nohighlight">\(\boldsymbol{C}\)</span> is defined as</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] &amp; \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
\mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] &amp; \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\
\end{bmatrix},
\end{split}\]</div>
<p>where for example</p>
<div class="math notranslate nohighlight">
\[
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
\]</div>
<p>With this definition and recalling that the variance is defined as</p>
<div class="math notranslate nohighlight">
\[
\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
\]</div>
<p>we can rewrite the covariance matrix as</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] &amp; \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] &amp; \mathrm{var}[\boldsymbol{y}] \\
\end{bmatrix}.
\end{split}\]</div>
<p><strong>Note:</strong> we have used <span class="math notranslate nohighlight">\(1/n\)</span> in the above definitions of the <em>sample</em> variance and covariance. We assume then that we can calculate the exact mean value.
What you will find in essentially all statistics texts are equations
with a factor <span class="math notranslate nohighlight">\(1/(n-1)\)</span>. This is called <a class="reference external" href="https://mathworld.wolfram.com/BesselsCorrection.html">Bessel’s correction</a>. This
method corrects the bias in the estimation of the population variance
and covariance. It also partially corrects the bias in the estimation
of the population standard deviation. If you use a library like
<strong>Scikit-Learn</strong> or <strong>nunmpy’s</strong> function calculate the covariance, this
quantity will be computed with a factor <span class="math notranslate nohighlight">\(1/(n-1)\)</span>.</p>
<p>The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
large values. It is common to scale the covariance matrix by
introducing instead the correlation matrix defined via the so-called
correlation function</p>
<div class="math notranslate nohighlight">
\[
\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}.
\]</div>
<p>The correlation function is then given by values <span class="math notranslate nohighlight">\(\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]
\in [-1,1]\)</span>. This avoids eventual problems with too large values. We
can then define the correlation matrix for the two vectors <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>
and <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 &amp; \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\
\mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] &amp; 1 \\
\end{bmatrix},
\end{split}\]</div>
<p>In the above example this is the function we constructed using <strong>pandas</strong>.</p>
<p>In our derivation of the various regression algorithms like <strong>Ordinary Least Squares</strong> or <strong>Ridge regression</strong>
we defined the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> as</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X}=\begin{bmatrix}
x_{0,0} &amp; x_{0,1} &amp; x_{0,2}&amp; \dots &amp; \dots x_{0,p-1}\\
x_{1,0} &amp; x_{1,1} &amp; x_{1,2}&amp; \dots &amp; \dots x_{1,p-1}\\
x_{2,0} &amp; x_{2,1} &amp; x_{2,2}&amp; \dots &amp; \dots x_{2,p-1}\\
\dots &amp; \dots &amp; \dots &amp; \dots \dots &amp; \dots \\
x_{n-2,0} &amp; x_{n-2,1} &amp; x_{n-2,2}&amp; \dots &amp; \dots x_{n-2,p-1}\\
x_{n-1,0} &amp; x_{n-1,1} &amp; x_{n-1,2}&amp; \dots &amp; \dots x_{n-1,p-1}\\
\end{bmatrix},
\end{split}\]</div>
<p>with <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>, with the predictors/features <span class="math notranslate nohighlight">\(p\)</span> refering to the column numbers and the
entries <span class="math notranslate nohighlight">\(n\)</span> being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 &amp; \boldsymbol{x}_1 &amp; \boldsymbol{x}_2 &amp; \dots &amp; \dots &amp; \boldsymbol{x}_{p-1}\end{bmatrix},
\]</div>
<p>with a given vector</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} &amp; x_{1,i} &amp; x_{2,i}&amp; \dots &amp; \dots x_{n-1,i}\end{bmatrix}.
\]</div>
<p>With these definitions, we can now rewrite our <span class="math notranslate nohighlight">\(2\times 2\)</span>
correlation/covariance matrix in terms of a more general design/feature
matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>. This leads to a <span class="math notranslate nohighlight">\(p\times p\)</span>
covariance matrix for the vectors <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> with <span class="math notranslate nohighlight">\(i=0,1,\dots,p-1\)</span></p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
\mathrm{var}[\boldsymbol{x}_0] &amp; \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] &amp; \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] &amp; \dots &amp; \dots &amp; \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] &amp; \mathrm{var}[\boldsymbol{x}_1] &amp; \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] &amp; \dots &amp; \dots &amp; \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] &amp; \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] &amp; \mathrm{var}[\boldsymbol{x}_2] &amp; \dots &amp; \dots &amp; \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] &amp; \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] &amp; \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] &amp; \dots &amp; \dots &amp; \mathrm{var}[\boldsymbol{x}_{p-1}]\\
\end{bmatrix},
\end{split}\]</div>
<p>and the correlation matrix</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
1 &amp; \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] &amp; \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] &amp; \dots &amp; \dots &amp; \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] &amp; 1 &amp; \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] &amp; \dots &amp; \dots &amp; \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] &amp; \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] &amp; 1 &amp; \dots &amp; \dots &amp; \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
\dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] &amp; \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] &amp; \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] &amp; \dots &amp; \dots &amp; 1\\
\end{bmatrix},
\end{split}\]</div>
<p>The Numpy function <strong>np.cov</strong> calculates the covariance elements using
the factor <span class="math notranslate nohighlight">\(1/(n-1)\)</span> instead of <span class="math notranslate nohighlight">\(1/n\)</span> since it assumes we do not have
the exact mean values. The following simple function uses the
<strong>np.vstack</strong> function which takes each vector of dimension <span class="math notranslate nohighlight">\(1\times n\)</span>
and produces a <span class="math notranslate nohighlight">\(2\times n\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{W}\)</span></p>
<p>Note that this assumes you have the features as the rows, and the inputs as columns, that is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{W} = \begin{bmatrix} x_0 &amp; x_1 &amp; x_2 &amp; \dots &amp; x_{n-2} &amp; x_{n-1} \\
y_0 &amp; y_1 &amp; y_2 &amp; \dots &amp; y_{n-2} &amp; y_{n-1} \\
\end{bmatrix},
\end{split}\]</div>
<p>which in turn is converted into into the <span class="math notranslate nohighlight">\(2\times 2\)</span> covariance matrix
<span class="math notranslate nohighlight">\(\boldsymbol{C}\)</span> via the Numpy function <strong>np.cov()</strong>. We note that we can also calculate
the mean value of each set of samples <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> etc using the Numpy
function <strong>np.mean(x)</strong>. We can also extract the eigenvalues of the
covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Importing various packages</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y</span><span class="p">))</span>
<span class="n">W</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">vstack</span><span class="p">((</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">))</span>
<span class="n">C</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">cov</span><span class="p">(</span><span class="n">W</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">C</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.08873443359350565
3.7851533175757255
[[ 0.98248312 3.05483267]
[ 3.05483267 10.24784064]]
</pre></div>
</div>
</div>
</div>
<p>The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
subtract the mean values for each column. This leads to the following
code which sets up the correlations matrix for the previous example in
a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the <span class="math notranslate nohighlight">\(2\times 2\)</span> correlation matrix (since we have only two vectors).</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="c1"># define two vectors </span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
<span class="c1">#scaling the x and y vectors </span>
<span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">y</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="n">variance_x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">x</span><span class="nd">@x</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="n">variance_y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">y</span><span class="nd">@y</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="nb">print</span><span class="p">(</span><span class="n">variance_x</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">variance_y</span><span class="p">)</span>
<span class="n">cov_xy</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">x</span><span class="nd">@y</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="n">cov_xx</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">x</span><span class="nd">@x</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="n">cov_yy</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">y</span><span class="nd">@y</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="n">C</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">))</span>
<span class="n">C</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">]</span><span class="o">=</span> <span class="n">cov_xx</span><span class="o">/</span><span class="n">variance_x</span>
<span class="n">C</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span><span class="o">=</span> <span class="n">cov_yy</span><span class="o">/</span><span class="n">variance_y</span>
<span class="n">C</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span><span class="o">=</span> <span class="n">cov_xy</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">variance_y</span><span class="o">*</span><span class="n">variance_x</span><span class="p">)</span>
<span class="n">C</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">]</span><span class="o">=</span> <span class="n">C</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span>
<span class="nb">print</span><span class="p">(</span><span class="n">C</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07858099596662704
2.071920625289855
[[1. 0.71822416]
[0.71822416 1. ]]
</pre></div>
</div>
</div>
</div>
<p>We see that the matrix elements along the diagonal are one as they
should be and that the matrix is symmetric. Furthermore, diagonalizing
this matrix we easily see that it is a positive definite matrix.</p>
<p>The above procedure with <strong>numpy</strong> can be made more compact if we use <strong>pandas</strong>.</p>
<p>We know here how we can set up the correlation matrix using <strong>pandas</strong>, as done in this simple code</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">10</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">y</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="c1"># Note that we transpose the matrix in order to stay with our ordering n x p</span>
<span class="n">X</span> <span class="o">=</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">vstack</span><span class="p">((</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)))</span><span class="o">.</span><span class="n">T</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="n">Xpd</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">Xpd</span><span class="p">)</span>
<span class="n">correlation_matrix</span> <span class="o">=</span> <span class="n">Xpd</span><span class="o">.</span><span class="n">corr</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">correlation_matrix</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ -2.84861838 -10.07337358]
[ 0.53938383 2.59445979]
[ -0.40980089 -0.48871288]
[ 0.05834332 -0.39384255]
[ 2.25385387 7.58112299]
[ 0.68246434 2.46650488]
[ -0.25366775 -1.97047717]
[ 0.79081838 2.03807267]
[ -0.06150169 -0.57109235]
[ -0.75127504 -1.18266178]]
0 1
0 -2.848618 -10.073374
1 0.539384 2.594460
2 -0.409801 -0.488713
3 0.058343 -0.393843
4 2.253854 7.581123
5 0.682464 2.466505
6 -0.253668 -1.970477
7 0.790818 2.038073
8 -0.061502 -0.571092
9 -0.751275 -1.182662
0 1
0 1.000000 0.984525
1 0.984525 1.000000
</pre></div>
</div>
</div>
</div>
<p>We expand this model to the Franke function discussed earlier.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="k">def</span> <span class="nf">FrankeFunction</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">):</span>
<span class="n">term1</span> <span class="o">=</span> <span class="mf">0.75</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mf">0.25</span><span class="o">*</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">-</span> <span class="mf">0.25</span><span class="o">*</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
<span class="n">term2</span> <span class="o">=</span> <span class="mf">0.75</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="mf">49.0</span> <span class="o">-</span> <span class="mf">0.1</span><span class="o">*</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">+</span><span class="mi">1</span><span class="p">))</span>
<span class="n">term3</span> <span class="o">=</span> <span class="mf">0.5</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="o">/</span><span class="mf">4.0</span> <span class="o">-</span> <span class="mf">0.25</span><span class="o">*</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">3</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
<span class="n">term4</span> <span class="o">=</span> <span class="o">-</span><span class="mf">0.2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">4</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span> <span class="o">-</span> <span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
<span class="k">return</span> <span class="n">term1</span> <span class="o">+</span> <span class="n">term2</span> <span class="o">+</span> <span class="n">term3</span> <span class="o">+</span> <span class="n">term4</span>
<span class="k">def</span> <span class="nf">create_X</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">n</span> <span class="p">):</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ravel</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ravel</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="n">N</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">l</span> <span class="o">=</span> <span class="nb">int</span><span class="p">((</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">n</span><span class="o">+</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span> <span class="c1"># Number of elements in beta</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">N</span><span class="p">,</span><span class="n">l</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
<span class="n">q</span> <span class="o">=</span> <span class="nb">int</span><span class="p">((</span><span class="n">i</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
<span class="n">X</span><span class="p">[:,</span><span class="n">q</span><span class="o">+</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">i</span><span class="o">-</span><span class="n">k</span><span class="p">))</span><span class="o">*</span><span class="p">(</span><span class="n">y</span><span class="o">**</span><span class="n">k</span><span class="p">)</span>
<span class="k">return</span> <span class="n">X</span>
<span class="c1"># Making meshgrid of datapoints and compute Franke&#39;s function</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">N</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
<span class="n">z</span> <span class="o">=</span> <span class="n">FrankeFunction</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">create_X</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">n</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
<span class="n">Xpd</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="c1"># subtract the mean values and set up the covariance matrix</span>
<span class="n">Xpd</span> <span class="o">=</span> <span class="n">Xpd</span> <span class="o">-</span> <span class="n">Xpd</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="n">covariance_matrix</span> <span class="o">=</span> <span class="n">Xpd</span><span class="o">.</span><span class="n">cov</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">covariance_matrix</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.0 0.090241 0.082140 0.090564 0.084086 0.078082 0.082282 0.076619
2 0.0 0.082140 0.075227 0.083102 0.077428 0.072150 0.075982 0.070945
3 0.0 0.090564 0.083102 0.096893 0.090268 0.084107 0.091647 0.085571
4 0.0 0.084086 0.077428 0.090268 0.084286 0.078707 0.085655 0.080120
5 0.0 0.078082 0.072150 0.084107 0.078707 0.073657 0.080061 0.075020
6 0.0 0.082282 0.075982 0.091647 0.085655 0.080061 0.089082 0.083380
7 0.0 0.076619 0.070945 0.085571 0.080120 0.075020 0.083380 0.078158
8 0.0 0.071394 0.066284 0.079944 0.074984 0.070333 0.078082 0.073299
9 0.0 0.066569 0.061966 0.074729 0.070216 0.065973 0.073159 0.068776
10 0.0 0.073831 0.068541 0.084523 0.079224 0.074258 0.083779 0.078587
11 0.0 0.068867 0.064081 0.079021 0.074183 0.069640 0.078484 0.073716
12 0.0 0.064284 0.059952 0.073925 0.069506 0.065349 0.073567 0.069187
13 0.0 0.060048 0.056127 0.069202 0.065165 0.061359 0.068999 0.064974
14 0.0 0.056131 0.052581 0.064823 0.061133 0.057648 0.064753 0.061054
8 9 10 11 12 13 14
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.071394 0.066569 0.073831 0.068867 0.064284 0.060048 0.056131
2 0.066284 0.061966 0.068541 0.064081 0.059952 0.056127 0.052581
3 0.079944 0.074729 0.084523 0.079021 0.073925 0.069202 0.064823
4 0.074984 0.070216 0.079224 0.074183 0.069506 0.065165 0.061133
5 0.070333 0.065973 0.074258 0.069640 0.065349 0.061359 0.057648
6 0.078082 0.073159 0.083779 0.078484 0.073567 0.068999 0.064753
7 0.073299 0.068776 0.078587 0.073716 0.069187 0.064974 0.061054
8 0.068841 0.064684 0.073750 0.069268 0.065095 0.061209 0.057588
9 0.064684 0.060863 0.069242 0.065118 0.061272 0.057686 0.054340
10 0.073750 0.069242 0.079948 0.075028 0.070450 0.066189 0.062220
11 0.069268 0.065118 0.075028 0.070494 0.066270 0.062333 0.058663
12 0.065095 0.061272 0.070450 0.066270 0.062370 0.058732 0.055337
13 0.061209 0.057686 0.066189 0.062333 0.058732 0.055369 0.052227
14 0.057588 0.054340 0.062220 0.058663 0.055337 0.052227 0.049318
</pre></div>
</div>
</div>
</div>
<p>We note here that the covariance is zero for the first rows and
columns since all matrix elements in the design matrix were set to one
(we are fitting the function in terms of a polynomial of degree <span class="math notranslate nohighlight">\(n\)</span>).</p>
<p>This means that the variance for these elements will be zero and will
cause problems when we set up the correlation matrix. We can simply
drop these elements and construct a correlation
matrix without these elements.</p>
<p>We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}].
\]</div>
<p>To see this let us simply look at a design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{2\times 2}\)</span></p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X}=\begin{bmatrix}
x_{00} &amp; x_{01}\\
x_{10} &amp; x_{11}\\
\end{bmatrix}=\begin{bmatrix}
\boldsymbol{x}_{0} &amp; \boldsymbol{x}_{1}\\
\end{bmatrix}.
\end{split}\]</div>
<p>If we then compute the expectation value (note the <span class="math notranslate nohighlight">\(1/n\)</span> factor instead of <span class="math notranslate nohighlight">\(1/(n-1)\)</span>)</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\frac{1}{n}\begin{bmatrix}
x_{00}^2+x_{10}^2 &amp; x_{00}x_{01}+x_{10}x_{11}\\
x_{01}x_{00}+x_{11}x_{10} &amp; x_{01}^2+x_{11}^2\\
\end{bmatrix},
\end{split}\]</div>
<p>which is just</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] &amp; \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\
\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] &amp; \mathrm{var}[\boldsymbol{x}_1] \\
\end{bmatrix},
\end{split}\]</div>
<p>where we wrote <span class="math notranslate nohighlight">\(\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1]=\boldsymbol{C}[\boldsymbol{x}]\)</span> to indicate
that this is the covariance of the vectors <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> of the
design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>It is easy to generalize this to a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>.</p>
</div>
<div class="section" id="linking-with-the-svd">
<h2><span class="section-number">4.9. </span>Linking with the SVD<a class="headerlink" href="#linking-with-the-svd" title="Permalink to this headline"></a></h2>
<p>We saw earlier that</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
\]</div>
<p>Since the matrices here have dimension <span class="math notranslate nohighlight">\(p\times p\)</span>, with <span class="math notranslate nohighlight">\(p\)</span> corresponding to the singular values, we defined earlier the matrix</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} &amp; \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
\end{split}\]</div>
<p>where the tilde-matrix <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{\Sigma}}\)</span> is a matrix of dimension <span class="math notranslate nohighlight">\(p\times p\)</span> containing only the singular values <span class="math notranslate nohighlight">\(\sigma_i\)</span>, that is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 &amp; 0 &amp; 0 &amp; \dots &amp; 0 &amp; 0 \\
0 &amp; \sigma_1 &amp; 0 &amp; \dots &amp; 0 &amp; 0 \\
0 &amp; 0 &amp; \sigma_2 &amp; \dots &amp; 0 &amp; 0 \\
0 &amp; 0 &amp; 0 &amp; \dots &amp; \sigma_{p-2} &amp; 0 \\
0 &amp; 0 &amp; 0 &amp; \dots &amp; 0 &amp; \sigma_{p-1} \\
\end{bmatrix},
\end{split}\]</div>
<p>meaning we can write</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T.
\]</div>
<p>Multiplying from the right with <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> (using the orthogonality of <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>) we get</p>
<div class="math notranslate nohighlight">
\[
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2.
\]</div>
<p>This means the vectors <span class="math notranslate nohighlight">\(\boldsymbol{v}_i\)</span> of the orthogonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>
are the eigenvectors of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> with eigenvalues
given by the singular values squared, that is</p>
<div class="math notranslate nohighlight">
\[
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
\]</div>
<p>In other words, each non-zero singular value of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is a positive
square root of an eigenvalue of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>. It means also that
the columns of <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> are the eigenvectors of
<span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>. Since we have ordered the singular values of
<span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> in a descending order, it means that the column vectors
<span class="math notranslate nohighlight">\(\boldsymbol{v}_i\)</span> are hierarchically ordered by how much correlation they
encode from the columns of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>Note that these are also the eigenvectors and eigenvalues of the
Hessian matrix.</p>
<p>If we now recall the definition of the covariance matrix (not using
Bessel’s correction) we have</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X},
\]</div>
<p>meaning that every squared non-singular value of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> divided by <span class="math notranslate nohighlight">\(n\)</span> (
the number of samples) are the eigenvalues of the covariance
matrix. Every singular value of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is thus a positive square
root of an eigenvalue of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>. If the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is
self-adjoint, the singular values of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> are equal to the
absolute value of the eigenvalues of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>For <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T\)</span> we found</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T.
\]</div>
<p>Since the matrices here have dimension <span class="math notranslate nohighlight">\(n\times n\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} &amp; \boldsymbol{0} \\ \boldsymbol{0} &amp; \boldsymbol{0}\\ \end{bmatrix},
\end{split}\]</div>
<p>leading to</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} &amp; \boldsymbol{0} \\ \boldsymbol{0} &amp; \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T.
\end{split}\]</div>
<p>Multiplying with <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> from the right gives us the eigenvalue problem</p>
<div class="math notranslate nohighlight">
\[\begin{split}
(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} &amp; \boldsymbol{0} \\ \boldsymbol{0} &amp; \boldsymbol{0}\\ \end{bmatrix}.
\end{split}\]</div>
<p>It means that the eigenvalues of <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T\)</span> are again given by
the non-zero singular values plus now a series of zeros. The column
vectors of <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> are the eigenvectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T\)</span> and
measure how much correlations are contained in the rows of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
<p>Since we will mainly be interested in the correlations among the features
of our data (the columns of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>, the quantity of interest for us are the non-zero singular
values and the column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>.</p>
</div>
<div class="section" id="id1">
<h2><span class="section-number">4.10. </span>Ridge and Lasso Regression<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h2>
<p>Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is
our optimization problem is</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
\]</div>
<p>or we can state it as</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
\]</div>
<p>where we have used the definition of a norm-2 vector, that is</p>
<div class="math notranslate nohighlight">
\[
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
\]</div>
<p>By minimizing the above equation with respect to the parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we could then obtain an analytical expression for the
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. We can add a regularization parameter <span class="math notranslate nohighlight">\(\lambda\)</span> by
defining a new cost function to be optimized, that is</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
\]</div>
<p>which leads to the Ridge regression minimization problem where we
require that <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t\)</span>, where <span class="math notranslate nohighlight">\(t\)</span> is
a finite number larger than zero. By defining</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
\]</div>
<p>we have a new optimization equation</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \min_{\boldsymbol{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
\]</div>
<p>which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.</p>
<p>Here we have defined the norm-1 as</p>
<div class="math notranslate nohighlight">
\[
\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert.
\]</div>
<p>Using the matrix-vector expression for Ridge regression and dropping the parameter <span class="math notranslate nohighlight">\(1/n\)</span> in front of the standard means squared error equation, we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
\]</div>
<p>and
taking the derivatives with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we obtain then
a slightly modified matrix inversion problem which for finite values
of <span class="math notranslate nohighlight">\(\lambda\)</span> does not suffer from singularity problems. We obtain
the optimal parameters</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\]</div>
<p>with <span class="math notranslate nohighlight">\(\boldsymbol{I}\)</span> being a <span class="math notranslate nohighlight">\(p\times p\)</span> identity matrix with the constraint that</p>
<div class="math notranslate nohighlight">
\[
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
\]</div>
<p>with <span class="math notranslate nohighlight">\(t\)</span> a finite positive number.</p>
<p>When we compare this with the ordinary least squares result we have</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\]</div>
<p>which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>.</p>
<p>We see that Ridge regression is nothing but the standard OLS with a
modified diagonal term added to <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>. The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>, we may
even reduce the variance of the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.</p>
<p>Using our insights about the SVD of the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
\]</div>
<p>For Ridge regression this becomes</p>
<div class="math notranslate nohighlight">
\[
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
\]</div>
<p>with the vectors <span class="math notranslate nohighlight">\(\boldsymbol{u}_j\)</span> being the columns of <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> from the SVD of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. Note that the sums goes to <span class="math notranslate nohighlight">\(p-1\)</span> since.</p>
<p>Since <span class="math notranslate nohighlight">\(\lambda \geq 0\)</span>, it means that compared to OLS, we have</p>
<div class="math notranslate nohighlight">
\[
\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1.
\]</div>
<p>Ridge regression finds the coordinates of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with respect to the
orthonormal basis <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span>, it then shrinks the coordinates by
<span class="math notranslate nohighlight">\(\frac{\sigma_j^2}{\sigma_j^2+\lambda}\)</span>. Recall that the SVD has
eigenvalues ordered in a descending way, that is <span class="math notranslate nohighlight">\(\sigma_i \geq
\sigma_{i+1}\)</span>.</p>
<p>For small eigenvalues <span class="math notranslate nohighlight">\(\sigma_i\)</span> it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.</p>
<p>For the sake of simplicity, let us assume that the design matrix is orthonormal, that is</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}.
\]</div>
<p>In this case the standard OLS results in</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
\]</div>
<p>that is the Ridge estimator scales the OLS estimator by the inverse of a factor <span class="math notranslate nohighlight">\(1+\lambda\)</span>, and
the Ridge estimator converges to zero when the hyperparameter goes to
infinity.</p>
<p>We will come back to more interpreations after we have gone through some of the statistical analysis part.</p>
<p>Using the matrix-vector expression for Lasso regression and dropping the parameter <span class="math notranslate nohighlight">\(1/n\)</span> in front of the standard mean squared error equation, we have the following <strong>cost</strong> function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
\]</div>
<p>Taking the derivative with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 &amp; \beta &gt; 0 \\-1 &amp; \beta &lt; 0, \end{array}\right.
\end{split}\]</div>
<p>we have that the derivative of the cost function is</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
\]</div>
<p>and reordering we have</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a class="reference external" href="https://cvxopt.org/">CVXOPT</a>. We will discuss this later.</p>
<p>Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
diagonal. In this case we have an equal number of rows and columns <span class="math notranslate nohighlight">\(n=p\)</span>.</p>
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\beta}\)</span> and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
\]</div>
<p>and minimizing we have that</p>
<div class="math notranslate nohighlight">
\[
\hat{\beta}_i^{\mathrm{OLS}} = y_i.
\]</div>
<p>For Ridge regression our cost function is</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
\]</div>
<p>and minimizing we have that</p>
<div class="math notranslate nohighlight">
\[
\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}.
\]</div>
<p>For Lasso regression our cost function is</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
\]</div>
<p>and minimizing we have that</p>
<div class="math notranslate nohighlight">
\[
-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
\]</div>
<p>which leads to</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &amp;\mathrm{if} &amp; y_i&gt; \frac{\lambda}{2}\\
y_i+\frac{\lambda}{2} &amp;\mathrm{if} &amp; y_i&lt; -\frac{\lambda}{2}\\
0 &amp;\mathrm{if} &amp; \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
\end{split}\]</div>
<p>Plotting these results (<a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf">figure in handwritten notes for week 36</a>) shows clearly that Lasso regression suppresses (sets to zero) values of <span class="math notranslate nohighlight">\(\beta_i\)</span> for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>. Ridge regression reduces on the other hand the values of <span class="math notranslate nohighlight">\(\beta_i\)</span> as function of <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>As another example,
let us assume we have a data set with outputs/targets given by the vector</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{y}=\begin{bmatrix}4 \\ 2 \\3\end{bmatrix},
\end{split}\]</div>
<p>and our inputs as a <span class="math notranslate nohighlight">\(3\times 2\)</span> design matrix</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X}=\begin{bmatrix}2 &amp; 0\\ 0 &amp; 1 \\ 0 &amp; 0\end{bmatrix},
\end{split}\]</div>
<p>meaning that we have two features and two unknown parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> to be determined either by ordinary least squares, Ridge or Lasso regression.</p>
<p>For ordinary least squares (OLS) we know that the optimal solution is</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>Inserting the above values we obtain that</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
\end{split}\]</div>
<p>The code which implements this simpler case is presented after the discussion of Ridge and Lasso.</p>
<p>For Ridge regression we have</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>Inserting the above values we obtain that</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
\end{split}\]</div>
<p>There is normally a constraint on the value of <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\beta}\vert\vert_2\)</span> via the parameter <span class="math notranslate nohighlight">\(\lambda\)</span>.
Let us for simplicity assume that <span class="math notranslate nohighlight">\(\beta_0^2+\beta_1^2=1\)</span> as constraint. This will allow us to find an expression for the optimal values of <span class="math notranslate nohighlight">\(\beta\)</span> and <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>To see this, let us write the cost function for Ridge regression.</p>
<p>We define the MSE without the <span class="math notranslate nohighlight">\(1/n\)</span> factor and have then, using that</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix},
\end{split}\]</div>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2),
\]</div>
<p>and taking the derivative with respect to <span class="math notranslate nohighlight">\(\beta_0\)</span> we get</p>
<div class="math notranslate nohighlight">
\[
\beta_0=\frac{8}{4+\lambda},
\]</div>
<p>and for <span class="math notranslate nohighlight">\(\beta_1\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\beta_1=\frac{2}{1+\lambda},
\]</div>
<p>Using the constraint for <span class="math notranslate nohighlight">\(\beta_0^2+\beta_1^2=1\)</span> we can constrain <span class="math notranslate nohighlight">\(\lambda\)</span> by solving</p>
<div class="math notranslate nohighlight">
\[
\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1,
\]</div>
<p>which gives <span class="math notranslate nohighlight">\(\lambda=4.571\)</span> and <span class="math notranslate nohighlight">\(\beta_0=0.933\)</span> and <span class="math notranslate nohighlight">\(\beta_1=0.359\)</span>.</p>
<p>For Lasso we need now, keeping a constraint on <span class="math notranslate nohighlight">\(\vert\beta_0\vert+\vert\beta_1\vert=1\)</span>, to take the derivative of the absolute values of <span class="math notranslate nohighlight">\(\beta_0\)</span>
and <span class="math notranslate nohighlight">\(\beta_1\)</span>. This gives us the following derivatives of the cost function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
\]</div>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
\]</div>
<p>We have now four cases to solve besides the trivial cases <span class="math notranslate nohighlight">\(\beta_0\)</span> and/or <span class="math notranslate nohighlight">\(\beta_1\)</span> are zero, namely</p>
<ol class="simple">
<li><p><span class="math notranslate nohighlight">\(\beta_0 &gt; 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 &gt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\beta_0 &gt; 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 &lt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\beta_0 &lt; 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 &gt; 0\)</span>,</p></li>
<li><p><span class="math notranslate nohighlight">\(\beta_0 &lt; 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 &lt; 0\)</span>.</p></li>
</ol>
<p>If we consider the first case, we have then</p>
<div class="math notranslate nohighlight">
\[
-4(4-2\beta_0)+\lambda=0,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
-2(2-\beta_1)+\lambda=0.
\]</div>
<p>which yields</p>
<div class="math notranslate nohighlight">
\[
\beta_0=\frac{16+\lambda}{8},
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\beta_1=\frac{4+\lambda}{2}.
\]</div>
<p>Using the constraint on <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> we can then find the optimal value of <span class="math notranslate nohighlight">\(\lambda\)</span> for the different cases. We leave this as an exercise to you.</p>
<p>Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of <span class="math notranslate nohighlight">\(\lambda\)</span>, meaning that we need to perform a search in order to find the optimal values.</p>
<p>First we study and compare the OLS and Ridge results. The next code compares all three methods.
We select values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\in [10^{-4},10^4]\)</span> and compute the predicted values for ordinary least squares and Ridge regression.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="c1"># A seed just to ensure that the random numbers are the same for every run.</span>
<span class="c1"># Useful for eventual debugging.</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span> <span class="p">[</span> <span class="p">[</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">]])</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span>
<span class="c1"># matrix inversion to find beta</span>
<span class="n">OLSbeta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span><span class="p">)</span> <span class="o">@</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span>
<span class="nb">print</span><span class="p">(</span><span class="n">OLSbeta</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ytildeOLS</span> <span class="o">=</span> <span class="n">X</span> <span class="o">@</span> <span class="n">OLSbeta</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Training MSE for OLS&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">ytildeOLS</span><span class="p">))</span>
<span class="n">ypredictOLS</span> <span class="o">=</span> <span class="n">X</span> <span class="o">@</span> <span class="n">OLSbeta</span>
<span class="c1"># Repeat now for Ridge regression and various values of the regularization parameter</span>
<span class="n">I</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">eye</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="c1"># Decide which values of lambda to use</span>
<span class="n">nlambdas</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">MSEPredict</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">lambdas</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">logspace</span><span class="p">(</span><span class="o">-</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">nlambdas</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">):</span>
<span class="n">lmb</span> <span class="o">=</span> <span class="n">lambdas</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="n">Ridgebeta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span><span class="o">+</span><span class="n">lmb</span><span class="o">*</span><span class="n">I</span><span class="p">)</span> <span class="o">@</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span>
<span class="c1"># print(Ridgebeta)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">X</span> <span class="o">@</span> <span class="n">Ridgebeta</span>
<span class="n">MSEPredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span>
<span class="c1"># print(MSEPredict[i])</span>
<span class="c1"># Now plot the results</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSEPredict</span><span class="p">,</span> <span class="s1">&#39;r--&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Ridge Train&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">&#39;log10(lambda)&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;MSE&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[2. 2.]
Training MSE for OLS
3.0
</pre></div>
</div>
<img alt="_images/chapter2_252_1.png" src="_images/chapter2_252_1.png" />
</div>
</div>
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
<p>This happens also for Lasso regression, as seen from the next code
output. The difference is that Lasso shrinks the values of <span class="math notranslate nohighlight">\(\beta\)</span> to
zero at a much earlier stage and the results flatten out. We see that
Lasso gives also an excellent fit for small values of <span class="math notranslate nohighlight">\(\lambda\)</span> and
shows the best performance of the three regression methods.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="c1"># A seed just to ensure that the random numbers are the same for every run.</span>
<span class="c1"># Useful for eventual debugging.</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span> <span class="p">[</span> <span class="p">[</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">]])</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span>
<span class="c1"># matrix inversion to find beta</span>
<span class="n">OLSbeta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span><span class="p">)</span> <span class="o">@</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span>
<span class="nb">print</span><span class="p">(</span><span class="n">OLSbeta</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ytildeOLS</span> <span class="o">=</span> <span class="n">X</span> <span class="o">@</span> <span class="n">OLSbeta</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Training MSE for OLS&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">ytildeOLS</span><span class="p">))</span>
<span class="n">ypredictOLS</span> <span class="o">=</span> <span class="n">X</span> <span class="o">@</span> <span class="n">OLSbeta</span>
<span class="c1"># Repeat now for Ridge regression and various values of the regularization parameter</span>
<span class="n">I</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">eye</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="c1"># Decide which values of lambda to use</span>
<span class="n">nlambdas</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">MSERidgePredict</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">MSELassoPredict</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">lambdas</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">logspace</span><span class="p">(</span><span class="o">-</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">nlambdas</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">):</span>
<span class="n">lmb</span> <span class="o">=</span> <span class="n">lambdas</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="n">Ridgebeta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span><span class="o">+</span><span class="n">lmb</span><span class="o">*</span><span class="n">I</span><span class="p">)</span> <span class="o">@</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span>
<span class="nb">print</span><span class="p">(</span><span class="n">Ridgebeta</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">X</span> <span class="o">@</span> <span class="n">Ridgebeta</span>
<span class="n">MSERidgePredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span>
<span class="n">RegLasso</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Lasso</span><span class="p">(</span><span class="n">lmb</span><span class="p">)</span>
<span class="n">RegLasso</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="n">y</span><span class="p">)</span>
<span class="n">ypredictLasso</span> <span class="o">=</span> <span class="n">RegLasso</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">RegLasso</span><span class="o">.</span><span class="n">coef_</span><span class="p">)</span>
<span class="n">MSELassoPredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">ypredictLasso</span><span class="p">)</span>
<span class="c1"># Now plot the results</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSERidgePredict</span><span class="p">,</span> <span class="s1">&#39;r--&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Ridge Train&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSELassoPredict</span><span class="p">,</span> <span class="s1">&#39;r--&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Lasso Train&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">&#39;log10(lambda)&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;MSE&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[2. 2.]
Training MSE for OLS
3.0
[1.99995 1.99980002]
[ 0.50001525 -0.99953475]
[1.99993978 1.99975913]
[ 0.50001525 -0.99944272]
[1.99992746 1.99970988]
[ 0.50001525 -0.99933188]
[1.99991263 1.99965056]
[ 0.50001525 -0.99919837]
[1.99989476 1.99957911]
[ 0.50001524 -0.99903755]
[1.99987324 1.99949306]
[ 0.50001524 -0.99884384]
[1.99984732 1.99938942]
[ 0.50001524 -0.99861053]
[1.9998161 1.99926459]
[ 0.50001523 -0.9983295 ]
[1.99977849 1.99911427]
[ 0.50001523 -0.99799099]
[1.9997332 1.99893323]
[ 0.50001522 -0.99758326]
[1.99967865 1.99871521]
[ 0.50001521 -0.99709215]
[1.99961294 1.99845267]
[ 0.50001521 -0.99650061]
[1.99953381 1.99813653]
[ 0.50001519 -0.99578809]
[1.9994385 1.99775587]
[ 0.50001518 -0.99492986]
[1.9993237 1.99729756]
[ 0.50001517 -0.99389612]
[1.99918546 1.9967458 ]
[ 0.50001515 -0.99265097]
[1.99901896 1.99608161]
[ 0.50001512 -0.99115119]
[1.99881845 1.99528218]
[ 0.5000151 -0.9893447]
[1.998577 1.9943201]
[ 0.50001506 -0.98716878]
[1.99828624 1.99316252]
[ 0.50001502 -0.98454786]
[1.99793613 1.99176998]
[ 0.50001498 -0.98139097]
[1.99751458 1.99009525]
[ 0.50001492 -0.97758848]
[1.99700706 1.98808176]
[ 0.50001485 -0.97300836]
[1.9963961 1.98566191]
[ 0.50001476 -0.9674916 ]
[1.99566069 1.98275501]
[ 0.50001466 -0.96084663]
[1.9947756 1.97926491]
[ 0.50001454 -0.95284275]
[1.99371056 1.97507735]
[ 0.50001439 -0.94320205]
[1.99242921 1.97005689]
[ 0.50001422 -0.93158979]
[1.99088801 1.9640435 ]
[ 0.500014 -0.91760278]
[1.9890348 1.95684892]
[ 0.50001374 -0.90075537]
[1.98680716 1.9482527 ]
[ 0.50001343 -0.88046261]
[1.98413059 1.93799826]
[ 0.50001306 -0.85601992]
[1.98091621 1.92578916]
[ 0.50001261 -0.8265786 ]
[1.97705827 1.91128596]
[ 0.50001207 -0.79111643]
[1.97243128 1.89410423]
[ 0.50001142 -0.74840212]
[1.96688672 1.87381451]
[ 0.50001063 -0.69695259]
[1.96024953 1.84994524]
[ 0.50000969 -0.63498144]
[1.95231424 1.82198978]
[ 0.50000855 -0.56033697]
[1.94284104 1.78941903]
[ 0.50000718 -0.47042744]
[1.93155188 1.75170092]
[ 0.50000553 -0.3621311 ]
[1.91812702 1.70832814]
[ 0.50001414 -0.23167717]
[1.90220243 1.65885453]
[ 0.50000455 -0.07456491]
[1.88336879 1.60293962]
[ 0.47132891 -0. ]
[1.86117291 1.54039921]
[ 0.41433969 -0. ]
[1.83512277 1.47125748]
[ 0.34569596 -0. ]
[1.80469739 1.39579407]
[ 0.26301436 -0. ]
[1.76936315 1.31457796]
[ 0.16342407 -0. ]
[1.72859758 1.22847924]
[ 0.04346721 -0. ]
[1.68192193 1.13865173]
[ 0. -0.]
[1.62894215 1.04648335]
[ 0. -0.]
[1.56939714 0.95351665]
[ 0. -0.]
[1.50321091 0.86134827]
[ 0. -0.]
[1.43054282 0.77152076]
[ 0. -0.]
[1.35182854 0.68542204]
[ 0. -0.]
[1.26780278 0.60420593]
[ 0. -0.]
[1.17949575 0.52874252]
[ 0. -0.]
[1.0881981 0.45960079]
[ 0. -0.]
[0.99539415 0.39706038]
[ 0. -0.]
[0.90266948 0.34114547]
[ 0. -0.]
[0.81160425 0.29167186]
[ 0. -0.]
[0.7236674 0.24829908]
[ 0. -0.]
[0.64012627 0.21058097]
[ 0. -0.]
[0.56198284 0.17801022]
[ 0. -0.]
[0.48994188 0.15005476]
[ 0. -0.]
[0.42441033 0.12618549]
[ 0. -0.]
[0.3655222 0.10589577]
[ 0. -0.]
[0.31318084 0.08871404]
[ 0. -0.]
[0.26710969 0.07421084]
[ 0. -0.]
[0.22690428 0.06200174]
[ 0. -0.]
[0.19207979 0.0517473 ]
[ 0. -0.]
[0.16211139 0.04315108]
[ 0. -0.]
[0.13646574 0.0359565 ]
[ 0. -0.]
[0.11462415 0.02994311]
[ 0. -0.]
[0.09609807 0.02492265]
[ 0. -0.]
[0.08043851 0.02073509]
[ 0. -0.]
[0.06724062 0.01724499]
[ 0. -0.]
[0.05614483 0.01433809]
[ 0. -0.]
[0.04683565 0.01191824]
[ 0. -0.]
[0.039039 0.00990475]
[ 0. -0.]
[0.03251863 0.00823002]
[ 0. -0.]
[0.02707227 0.00683748]
[ 0. -0.]
[0.02252765 0.0056799 ]
[ 0. -0.]
[0.01873869 0.00471782]
[ 0. -0.]
[0.01558197 0.00391839]
[ 0. -0.]
[0.01295356 0.0032542 ]
[ 0. -0.]
[0.01076611 0.00270244]
[ 0. -0.]
[0.00894639 0.00224413]
[ 0. -0.]
[0.0074331 0.00186347]
[ 0. -0.]
[0.00617499 0.00154733]
[ 0. -0.]
[0.00512927 0.00128479]
[ 0. -0.]
[0.00426027 0.00106677]
[ 0. -0.]
[0.00353823 0.00088573]
[ 0. -0.]
[0.00293838 0.00073541]
[ 0. -0.]
[0.0024401 0.00061058]
[ 0. -0.]
[0.00202624 0.00050694]
[ 0. -0.]
[0.00168251 0.00042089]
[ 0. -0.]
[0.00139705 0.00034944]
[ 0. -0.]
[0.00115999 0.00029012]
[ 0. -0.]
[0.00096314 0.00024087]
[ 0. -0.]
[0.00079968 0.00019998]
[ 0. -0.]
</pre></div>
</div>
<img alt="_images/chapter2_254_1.png" src="_images/chapter2_254_1.png" />
</div>
</div>
<p>We bring then back our exponential function example and study all
three regression methods. Depending on the level of noise, we note
that for small values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span> all three
methods produce the same mean squared error. Again, Lasso shrinks the
parameter values to zero much earlier than Ridge regression and the
Lasso results flatten out much earlier since all <span class="math notranslate nohighlight">\(\beta_j=0\)</span> (check
this by printing the values). This case is an example of where OLS
performs best. Lasso and Ridge reproduce the OLS results for a limited
set of <span class="math notranslate nohighlight">\(\lambda\)</span> values.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="c1"># A seed just to ensure that the random numbers are the same for every run.</span>
<span class="c1"># Useful for eventual debugging.</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">3155</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="c1"># number of features p (here degree of polynomial</span>
<span class="n">p</span> <span class="o">=</span> <span class="mi">3</span>
<span class="c1"># The design matrix now as function of a given polynomial</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="n">p</span><span class="p">))</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span>
<span class="c1"># We split the data in test and training data</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="c1"># matrix inversion to find beta</span>
<span class="n">OLSbeta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X_train</span><span class="p">)</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y_train</span>
<span class="nb">print</span><span class="p">(</span><span class="n">OLSbeta</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ytildeOLS</span> <span class="o">=</span> <span class="n">X_train</span> <span class="o">@</span> <span class="n">OLSbeta</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Training MSE for OLS&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytildeOLS</span><span class="p">))</span>
<span class="n">ypredictOLS</span> <span class="o">=</span> <span class="n">X_test</span> <span class="o">@</span> <span class="n">OLSbeta</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test MSE OLS&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictOLS</span><span class="p">))</span>
<span class="c1"># Repeat now for Lasso and Ridge regression and various values of the regularization parameter</span>
<span class="n">I</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">eye</span><span class="p">(</span><span class="n">p</span><span class="p">,</span><span class="n">p</span><span class="p">)</span>
<span class="c1"># Decide which values of lambda to use</span>
<span class="n">nlambdas</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">MSEPredict</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">MSETrain</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">MSELassoPredict</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">MSELassoTrain</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">lambdas</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">logspace</span><span class="p">(</span><span class="o">-</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="n">nlambdas</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">):</span>
<span class="n">lmb</span> <span class="o">=</span> <span class="n">lambdas</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="n">Ridgebeta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">+</span><span class="n">lmb</span><span class="o">*</span><span class="n">I</span><span class="p">)</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y_train</span>
<span class="c1"># include lasso using Scikit-Learn</span>
<span class="n">RegLasso</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Lasso</span><span class="p">(</span><span class="n">lmb</span><span class="p">)</span>
<span class="n">RegLasso</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">y_train</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ytildeRidge</span> <span class="o">=</span> <span class="n">X_train</span> <span class="o">@</span> <span class="n">Ridgebeta</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">X_test</span> <span class="o">@</span> <span class="n">Ridgebeta</span>
<span class="n">ytildeLasso</span> <span class="o">=</span> <span class="n">RegLasso</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">ypredictLasso</span> <span class="o">=</span> <span class="n">RegLasso</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">MSEPredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span>
<span class="n">MSETrain</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytildeRidge</span><span class="p">)</span>
<span class="n">MSELassoPredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictLasso</span><span class="p">)</span>
<span class="n">MSELassoTrain</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytildeLasso</span><span class="p">)</span>
<span class="c1"># Now plot the results</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSETrain</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Ridge train&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSEPredict</span><span class="p">,</span> <span class="s1">&#39;r--&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Ridge Test&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSELassoTrain</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Lasso train&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSELassoPredict</span><span class="p">,</span> <span class="s1">&#39;r--&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Lasso Test&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">&#39;log10(lambda)&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;MSE&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.03099776 -0.17917768 5.18029127]
Training MSE for OLS
0.009163470508352218
Test MSE OLS
0.008675369724975977
</pre></div>
</div>
<img alt="_images/chapter2_256_1.png" src="_images/chapter2_256_1.png" />
</div>
</div>
<p>Both these example send a clear message. The addition of a
shrinkage/regularization term implies that we need to perform a search
for the optimal values of <span class="math notranslate nohighlight">\(\lambda\)</span>. We will see this throughout these
series of lectures.</p>
<p>As a small addendum, we note that you can also solve this problem using the convex optimization package <a class="reference external" href="https://cvxopt.org/examples/mlbook/l1regls.html">CVXOPT</a>. This requires, in addition to having installed <strong>CVXOPT</strong>, you need to download the file <em><a class="reference external" href="http://l1regl.py">l1regl.py</a></em>.</p>
</div>
<div class="section" id="linking-the-regression-analysis-with-a-statistical-interpretation">
<h2><span class="section-number">4.11. </span>Linking the regression analysis with a statistical interpretation<a class="headerlink" href="#linking-the-regression-analysis-with-a-statistical-interpretation" title="Permalink to this headline"></a></h2>
<p>We will now couple the discussions of ordinary least squares, Ridge
and Lasso regression with a statistical interpretation, that is we
move from a linear algebra analysis to a statistical analysis. In
particular, we will focus on what the regularization terms can result
in. We will amongst other things show that the regularization
parameter can reduce considerably the variance of the parameters
<span class="math notranslate nohighlight">\(\beta\)</span>.</p>
<p>The
advantage of doing linear regression is that we actually end up with
analytical expressions for several statistical quantities.<br />
Standard least squares and Ridge regression allow us to
derive quantities like the variance and other expectation values in a
rather straightforward way.</p>
<p>It is assumed that <span class="math notranslate nohighlight">\(\varepsilon_i
\sim \mathcal{N}(0, \sigma^2)\)</span> and the <span class="math notranslate nohighlight">\(\varepsilon_{i}\)</span> are
independent, i.e.:</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
\mbox{Cov}(\varepsilon_{i_1},
\varepsilon_{i_2}) &amp; = \left\{ \begin{array}{lcc} \sigma^2 &amp; \mbox{if}
&amp; i_1 = i_2, \\ 0 &amp; \mbox{if} &amp; i_1 \not= i_2. \end{array} \right.
\end{align*}
\end{split}\]</div>
<p>The randomness of <span class="math notranslate nohighlight">\(\varepsilon_i\)</span> implies that
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is also a random variable. In particular,
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is normally distributed, because <span class="math notranslate nohighlight">\(\varepsilon_i \sim
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\)</span> is a
non-random scalar. To specify the parameters of the distribution of
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> we need to calculate its first two moments.</p>
<p>Recall that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is a matrix of dimensionality <span class="math notranslate nohighlight">\(n\times p\)</span>. The
notation above <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast}\)</span> means that we are looking at the
row number <span class="math notranslate nohighlight">\(i\)</span> and perform a sum over all values <span class="math notranslate nohighlight">\(p\)</span>.</p>
<p>The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
that there exists a function <span class="math notranslate nohighlight">\(f(\boldsymbol{x})\)</span> and a normal distributed error <span class="math notranslate nohighlight">\(\boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2)\)</span>
which describe our data</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon}
\]</div>
<p>We approximate this function with our model from the solution of the linear regression equations, that is our
function <span class="math notranslate nohighlight">\(f\)</span> is approximated by <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> where we want to minimize <span class="math notranslate nohighlight">\((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\)</span>, our MSE, with</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
\]</div>
<p>We can calculate the expectation value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> for a given element <span class="math notranslate nohighlight">\(i\)</span></p>
<div class="math notranslate nohighlight">
\[
\begin{align*}
\mathbb{E}(y_i) &amp; =
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
\end{align*}
\]</div>
<p>while
its variance is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*} \mbox{Var}(y_i) &amp; = \mathbb{E} \{ [y_i
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
[\mathbb{E}(y_i)]^2 \\ &amp; = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &amp;
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
\ast} \, \beta)^2 \\ &amp; = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
\\ &amp; = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
\end{align*}
\end{split}\]</div>
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\beta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we can evaluate the expectation value</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
\]</div>
<p>This means that the estimator of the regression parameters is unbiased.</p>
<p>We can also calculate the variance</p>
<p>The variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{eqnarray*}
\mbox{Var}(\boldsymbol{\beta}) &amp; = &amp; \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
\\
&amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
\\
% &amp; = &amp; \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
% \\
% &amp; = &amp; \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
% \\
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
\\
&amp; = &amp; (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
% \\
% &amp; = &amp; (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
% \\
% &amp; &amp; + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
\\
&amp; = &amp; \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
\end{eqnarray*}
\end{split}\]</div>
<p>where we have used that <span class="math notranslate nohighlight">\(\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
\, (\mathbf{X}^{T} \mathbf{X})^{-1}\)</span>, one obtains an estimate of the
variance of the estimate of the <span class="math notranslate nohighlight">\(j\)</span>-th regression coefficient:
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
construct a confidence interval for the estimates.</p>
<p>In a similar way, we can obtain analytical expressions for say the
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and their variance
when we employ Ridge regression, allowing us again to define a confidence interval.</p>
<p>It is rather straightforward to show that</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
\]</div>
<p>We see clearly that
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span>. We say then that the ridge estimator is biased.</p>
<p>We can also compute the variance as</p>
<div class="math notranslate nohighlight">
\[
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
\]</div>
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> goes to zero.</p>
<p>With this, we can compute the difference</p>
<div class="math notranslate nohighlight">
\[
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
\]</div>
<p>The difference is non-negative definite since each component of the
matrix product is non-negative definite.
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda &gt; 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
</div>
<div class="section" id="deriving-ols-from-a-probability-distribution">
<h2><span class="section-number">4.12. </span>Deriving OLS from a probability distribution<a class="headerlink" href="#deriving-ols-from-a-probability-distribution" title="Permalink to this headline"></a></h2>
<p>Our basic assumption when we derived the OLS equations was to assume
that our output is determined by a given continuous function
<span class="math notranslate nohighlight">\(f(\boldsymbol{x})\)</span> and a random noise <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> given by the normal
distribution with zero mean value and an undetermined variance
<span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
<p>We found above that the outputs <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> have a mean value given by
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
the design matrix are not stochastic variables, we can assume that the
probability distribution of our targets is also a normal distribution
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span>. This means that a
single output <span class="math notranslate nohighlight">\(y_i\)</span> is given by the Gaussian distribution</p>
<div class="math notranslate nohighlight">
\[
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
\]</div>
<p>We assume now that the various <span class="math notranslate nohighlight">\(y_i\)</span> values are stochastically distributed according to the above Gaussian distribution.
We define this distribution as</p>
<div class="math notranslate nohighlight">
\[
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
\]</div>
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<p>Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
\]</div>
<p>We will write this in a more compact form reserving <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> for the domain of events, including the ouputs (targets) and the inputs. That is
in case we have a simple one-dimensional input and output case</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})].
\]</div>
<p>In the more general case the various inputs should be replaced by the possible features represented by the input data set <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.
We can now rewrite the above probability as</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
\]</div>
<p>It is a conditional probability (see below) and reads as the
likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<p>In statistics, maximum likelihood estimation (MLE) is a method of
estimating the parameters of an assumed probability distribution,
given some observed data. This is achieved by maximizing a likelihood
function so that, under the assumed statistical model, the observed
data is the most probable.</p>
<p>We will assume here that our events are given by the above Gaussian
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> by
maximizing the above PDF. However, computing the derivatives of a
product function is cumbersome and can easily lead to overflow and/or
underflowproblems, with potentials for loss of numerical precision.</p>
<p>In practice, it is more convenient to maximize the logarithm of the
PDF because it is a monotonically increasing function of the argument.
Alternatively, and this will be our option, we will minimize the
negative of the logarithm since this is a monotonically decreasing
function.</p>
<p>Note also that maximization/minimization of the logarithm of the PDF
is equivalent to the maximization/minimization of the function itself.</p>
<p>We could now define a new cost function to minimize, namely the negative logarithm of the above PDF</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
\]</div>
<p>which becomes</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
\]</div>
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\beta\)</span> we recognize our familiar OLS equation, namely</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
\]</div>
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\beta\)</span></p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
\]</div>
<p>Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.</p>
<p>A central theorem in statistics is Bayes’ theorem. This theorem plays a similar role as the good old Pythagoras’ theorem in geometry.
Bayes’ theorem is extremely simple to derive. But to do so we need some basic axioms from statistics.</p>
<p>Assume we have two domains of events <span class="math notranslate nohighlight">\(X=[x_0,x_1,\dots,x_{n-1}]\)</span> and <span class="math notranslate nohighlight">\(Y=[y_0,y_1,\dots,y_{n-1}]\)</span>.</p>
<p>We define also the likelihood for <span class="math notranslate nohighlight">\(X\)</span> and <span class="math notranslate nohighlight">\(Y\)</span> as <span class="math notranslate nohighlight">\(p(X)\)</span> and <span class="math notranslate nohighlight">\(p(Y)\)</span> respectively.
The likelihood of a specific event <span class="math notranslate nohighlight">\(x_i\)</span> (or <span class="math notranslate nohighlight">\(y_i\)</span>) is then written as <span class="math notranslate nohighlight">\(p(X=x_i)\)</span> or just <span class="math notranslate nohighlight">\(p(x_i)=p_i\)</span>.</p>
<p>The union of events is given by</p>
<div class="math notranslate nohighlight">
\[
p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
\]</div>
<p>The product rule (aka joint probability) is given by</p>
<div class="math notranslate nohighlight">
\[
p(X \cap Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
\]</div>
<p>where we read <span class="math notranslate nohighlight">\(p(X\vert Y)\)</span> as the likelihood of obtaining <span class="math notranslate nohighlight">\(X\)</span> given <span class="math notranslate nohighlight">\(Y\)</span>.</p>
<p>If we have independent events then <span class="math notranslate nohighlight">\(p(X,Y)=p(X)p(Y)\)</span>.</p>
<p>The marginal probability is defined in terms of only one of the set of variables <span class="math notranslate nohighlight">\(X,Y\)</span>. For a discrete probability we have</p>
<div class="math notranslate nohighlight">
\[
p(X)=\sum_{i=0}^{n-1}p(X,Y=y_i)=\sum_{i=0}^{n-1}p(X\vert Y=y_i)p(Y=y_i)=\sum_{i=0}^{n-1}p(X\vert y_i)p(y_i).
\]</div>
<p>The conditional probability, if <span class="math notranslate nohighlight">\(p(Y) &gt; 0\)</span>, is</p>
<div class="math notranslate nohighlight">
\[
p(X\vert Y)= \frac{p(X,Y)}{p(Y)}=\frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}.
\]</div>
<p>If we combine the conditional probability with the marginal probability and the standard product rule, we have</p>
<div class="math notranslate nohighlight">
\[
p(X\vert Y)= \frac{p(X,Y)}{p(Y)},
\]</div>
<p>which we can rewrite as</p>
<div class="math notranslate nohighlight">
\[
p(X\vert Y)= \frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}=\frac{p(Y\vert X)p(X)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)},
\]</div>
<p>which is Bayes’ theorem. It allows us to evaluate the uncertainty in in <span class="math notranslate nohighlight">\(X\)</span> after we have observed <span class="math notranslate nohighlight">\(Y\)</span>. We can easily interchange <span class="math notranslate nohighlight">\(X\)</span> with <span class="math notranslate nohighlight">\(Y\)</span>.</p>
<p>The quantity <span class="math notranslate nohighlight">\(p(Y\vert X)\)</span> on the right-hand side of the theorem is
evaluated for the observed data <span class="math notranslate nohighlight">\(Y\)</span> and can be viewed as a function of
the parameter space represented by <span class="math notranslate nohighlight">\(X\)</span>. This function is not
necesseraly normalized and is normally called the likelihood function.</p>
<p>The function <span class="math notranslate nohighlight">\(p(X)\)</span> on the right hand side is called the prior while
the function on the left hand side is the called the posterior
probability. The denominator on the right hand side serves as a
normalization factor for the posterior distribution.</p>
<p>Let us try to illustrate Bayes’ theorem through an example.</p>
<p>Let us suppose that you are undergoing a series of mammography scans
in order to rule out possible breast cancer cases. We define the
sensitivity for a positive event by the variable <span class="math notranslate nohighlight">\(X\)</span>. It takes binary
values with <span class="math notranslate nohighlight">\(X=1\)</span> representing a positive event and <span class="math notranslate nohighlight">\(X=0\)</span> being a
negative event. We reserve <span class="math notranslate nohighlight">\(Y\)</span> as a classification parameter for
either a negative or a positive breast cancer confirmation. (Short
note on wordings: positive here means having breast cancer, although
none of us would consider this being a positive thing).</p>
<p>We let <span class="math notranslate nohighlight">\(Y=1\)</span> represent the the case of having breast cancer and <span class="math notranslate nohighlight">\(Y=0\)</span> as not.</p>
<p>Let us assume that if you have breast cancer, the test will be positive with a probability of <span class="math notranslate nohighlight">\(0.8\)</span> (the numbers here are all made up),
that is we have</p>
<div class="math notranslate nohighlight">
\[
p(X=1\vert Y=1) =0.8.
\]</div>
<p>This obviously sounds scary since many would conclude that if the test
is positive, there is a likelihood of <span class="math notranslate nohighlight">\(80\%\)</span> for having cancer. It is
however not correct, as the following Bayesian analysis shows. The correct question to pose is <em>what is the probability of having breast cancer in case of a positive test?</em>
We are thus interested in</p>
<div class="math notranslate nohighlight">
\[
p(Y=1\vert X=1),
\]</div>
<p>instead of <span class="math notranslate nohighlight">\(p(X=1\vert Y=1)\)</span>.</p>
<p>If we look at various national surveys on breast cancer, the general
likelihood of developing breast cancer is a very small number. Let us
assume that the prior probability in the population as a whole is</p>
<div class="math notranslate nohighlight">
\[
p(Y=1) =0.004.
\]</div>
<p>We need also to account for the fact that the test may produce a false
positive result (false alarm). Let us here assume that we have</p>
<div class="math notranslate nohighlight">
\[
p(X=1\vert Y=0) =0.1.
\]</div>
<p>Using Bayes’ theorem we can then find the posterior probability that
the person has breast cancer in case of a positive test, that is we
can compute</p>
<!-- Equation labels as ordinary links -->
<div id="_auto2"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}
p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=
\label{_auto2} \tag{2}
\end{equation}
\]</div>
<!-- Equation labels as ordinary links -->
<div id="_auto3"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}
\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031.
\label{_auto3} \tag{3}
\end{equation}
\]</div>
<p>That is, in case of a positive test, there is only a <span class="math notranslate nohighlight">\(3\%\)</span> chance of having breast cancer!</p>
</div>
<div class="section" id="bayes-theorem-and-ridge-and-lasso-regression">
<h2><span class="section-number">4.13. </span>Bayes’ Theorem and Ridge and Lasso Regression<a class="headerlink" href="#bayes-theorem-and-ridge-and-lasso-regression" title="Permalink to this headline"></a></h2>
<p>Hitherto we have discussed Ridge and Lasso regression in terms of a
linear analysis. This may to many of you feel rather technical and
perhaps not that intuitive. The question is whether we can develop a
more intuitive way of understanding what Ridge and Lasso express.</p>
<p>Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.</p>
<p>We will play around with a study of the values for the optimal
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> using OLS, Ridge and Lasso regression. For
OLS, you will notice as function of the noise and polynomial degree,
that the parameters <span class="math notranslate nohighlight">\(\beta\)</span> will fluctuate from order to order in the
polynomial fit and that for larger and larger polynomial degrees of
freedom, the parameters will tend to increase in value for OLS.</p>
<p>For Ridge and Lasso regression, the higher order parameters will
typically be reduced, providing thereby less fluctuations from one
order to another one.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="c1"># Make data set.</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">10000</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">Maxpolydegree</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="n">Maxpolydegree</span><span class="p">))</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
<span class="k">for</span> <span class="n">polydegree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">Maxpolydegree</span><span class="p">):</span>
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">polydegree</span><span class="p">):</span>
<span class="n">X</span><span class="p">[:,</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">degree</span><span class="p">)</span>
<span class="c1"># We split the data in test and training data</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="c1"># matrix inversion to find beta</span>
<span class="n">OLSbeta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X_train</span><span class="p">)</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y_train</span>
<span class="nb">print</span><span class="p">(</span><span class="n">OLSbeta</span><span class="p">)</span>
<span class="n">ypredictOLS</span> <span class="o">=</span> <span class="n">X_test</span> <span class="o">@</span> <span class="n">OLSbeta</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test MSE OLS&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictOLS</span><span class="p">))</span>
<span class="c1"># Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn</span>
<span class="c1"># Decide which values of lambda to use</span>
<span class="n">nlambdas</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">MSERidgePredict</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">MSELassoPredict</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">lambdas</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">logspace</span><span class="p">(</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">nlambdas</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">):</span>
<span class="n">lmb</span> <span class="o">=</span> <span class="n">lambdas</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="c1"># Make the fit using Ridge and Lasso</span>
<span class="n">RegRidge</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Ridge</span><span class="p">(</span><span class="n">lmb</span><span class="p">,</span><span class="n">fit_intercept</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">RegRidge</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">y_train</span><span class="p">)</span>
<span class="n">RegLasso</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Lasso</span><span class="p">(</span><span class="n">lmb</span><span class="p">,</span><span class="n">fit_intercept</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">RegLasso</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">y_train</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">RegRidge</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">ypredictLasso</span> <span class="o">=</span> <span class="n">RegLasso</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="c1"># Compute the MSE and print it</span>
<span class="n">MSERidgePredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span>
<span class="n">MSELassoPredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictLasso</span><span class="p">)</span>
<span class="c1"># print(lmb,RegRidge.coef_)</span>
<span class="c1"># print(lmb,RegLasso.coef_)</span>
<span class="c1"># Now plot the results</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSERidgePredict</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Ridge Test&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSELassoPredict</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Lasso Test&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">&#39;log10(lambda)&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;MSE&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.0169643 0.27924636 -1.4087793 1.03308408 0. ]
Test MSE OLS
0.9582286166520774
</pre></div>
</div>
<img alt="_images/chapter2_322_1.png" src="_images/chapter2_322_1.png" />
</div>
</div>
<p>How can we understand this?</p>
<p>Let us write out the values of the coefficients <span class="math notranslate nohighlight">\(\beta_i\)</span> as functions
of the polynomial degree and noise. We will focus only on the Ridge
results and some few selected values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>If we don’t include any noise and run this code for different values
of the polynomial degree, we notice that the results for <span class="math notranslate nohighlight">\(\beta_i\)</span> do
not show great changes from one order to the next. This is an
indication that for higher polynomial orders, our parameters become
less important.</p>
<p>If we however add noise, what happens is that the polynomial fit is
trying to adjust the fit to traverse in the best possible way all data
points. This can lead to large fluctuations in the parameters
<span class="math notranslate nohighlight">\(\beta_i\)</span> as functions of polynomial order. It will also be reflected
in a larger value of the variance of each parameter <span class="math notranslate nohighlight">\(\beta_i\)</span>. What
Ridge regression (and Lasso as well) are doing then is to try to
quench the fluctuations in the parameters of <span class="math notranslate nohighlight">\(\beta_i\)</span> which have a
large variance (normally for higher orders in the polynomial).</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">display</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="c1"># Make data set.</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">Maxpolydegree</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="n">Maxpolydegree</span><span class="p">))</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
<span class="k">for</span> <span class="n">polydegree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">Maxpolydegree</span><span class="p">):</span>
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">polydegree</span><span class="p">):</span>
<span class="n">X</span><span class="p">[:,</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">degree</span><span class="p">)</span>
<span class="c1"># We split the data in test and training data</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="c1"># Decide which values of lambda to use</span>
<span class="n">nlambdas</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">lambdas</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">logspace</span><span class="p">(</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">nlambdas</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">):</span>
<span class="n">lmb</span> <span class="o">=</span> <span class="n">lambdas</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="c1"># Make the fit using Ridge only</span>
<span class="n">RegRidge</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Ridge</span><span class="p">(</span><span class="n">lmb</span><span class="p">,</span><span class="n">fit_intercept</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">RegRidge</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">y_train</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">RegRidge</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">Coeffs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">RegRidge</span><span class="o">.</span><span class="n">coef_</span><span class="p">)</span>
<span class="n">BetaValues</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">Coeffs</span><span class="p">)</span>
<span class="n">BetaValues</span><span class="o">.</span><span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;beta&#39;</span><span class="p">]</span>
<span class="n">display</span><span class="p">(</span><span class="n">BetaValues</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.986699</td>
</tr>
<tr>
<th>1</th>
<td>-0.606760</td>
</tr>
<tr>
<th>2</th>
<td>1.280573</td>
</tr>
<tr>
<th>3</th>
<td>-0.850164</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div><div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.978553</td>
</tr>
<tr>
<th>1</th>
<td>-0.511888</td>
</tr>
<tr>
<th>2</th>
<td>1.051418</td>
</tr>
<tr>
<th>3</th>
<td>-0.701370</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div><div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.946957</td>
</tr>
<tr>
<th>1</th>
<td>-0.162246</td>
</tr>
<tr>
<th>2</th>
<td>0.221921</td>
</tr>
<tr>
<th>3</th>
<td>-0.167787</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div><div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.906747</td>
</tr>
<tr>
<th>1</th>
<td>0.017665</td>
</tr>
<tr>
<th>2</th>
<td>-0.029483</td>
</tr>
<tr>
<th>3</th>
<td>-0.053849</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div><div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>beta</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.718165</td>
</tr>
<tr>
<th>1</th>
<td>0.156956</td>
</tr>
<tr>
<th>2</th>
<td>0.040102</td>
</tr>
<tr>
<th>3</th>
<td>-0.001880</td>
</tr>
<tr>
<th>4</th>
<td>0.000000</td>
</tr>
</tbody>
</table>
</div></div></div>
</div>
<p>As an exercise, repeat these calculations with ordinary least squares
only with and without noise. Calculate thereafter the variance of the
parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> as function of polynomial order and of the added
noise. Here we recommend to use <span class="math notranslate nohighlight">\(\sigma^2=1\)</span> as variance for the
added noise (which follows a normal distribution with mean value zero).
Comment your results. If you have a large noise term, do the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> vary more as function
of model complexity? And what about their variance?</p>
</div>
<div class="section" id="linking-bayes-theorem-with-ridge-and-lasso-regression">
<h2><span class="section-number">4.14. </span>Linking Bayes’ Theorem with Ridge and Lasso Regression<a class="headerlink" href="#linking-bayes-theorem-with-ridge-and-lasso-regression" title="Permalink to this headline"></a></h2>
<p>We have seen that Ridge regression suppresses those features which
have a small singular value. This corresponds to a feature which exhibits
a large variance in the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span>.
Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use
Bayes’ theorem in order to deepen our understanding of Ridge and Lasso regression.</p>
<p>For ordinary least squares we postulated that the maximum likelihood for the domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> (one-dimensional case)</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})],
\]</div>
<p>is given by</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
\]</div>
<p>In Bayes’ theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> given a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span>? That is, how can we define the posterior probability</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta}\vert\boldsymbol{D}).
\]</div>
<p>Bayes’ theorem comes to our rescue here since (omitting the normalization constant)</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}).
\]</div>
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\beta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\beta})\)</span>!</p>
<p>With the posterior probability defined by a likelihood which we have
already modeled and an unknown prior, we are now ready to make
additional models for the prior.</p>
<p>We can, based on our discussions of the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and
the mean value, assume that the prior for the values <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is
given by a Gaussian with mean value zero and variance <span class="math notranslate nohighlight">\(\tau^2\)</span>, that</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
\]</div>
<p>Our posterior probability becomes then (omitting the normalization factor which is just a constant)</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
\]</div>
<p>We can now optimize this quantity with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. As we
did for OLS, this is most conveniently done by taking the negative
logarithm of the posterior probability. Doing so and leaving out the
terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
\]</div>
<p>and replacing <span class="math notranslate nohighlight">\(1/2\tau^2\)</span> with <span class="math notranslate nohighlight">\(\lambda\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
\]</div>
<p>which is our Ridge cost function! Nice, isn’t it?</p>
<p>To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (<a class="reference external" href="https://en.wikipedia.org/wiki/Laplace_distribution">Laplace in this case</a>) with zero mean value, that is</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}.
\]</div>
<p>Our posterior probability becomes then (omitting the normalization factor which is just a constant)</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}.
\]</div>
<p>Taking the negative
logarithm of the posterior probability and leaving out the
constants terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
\]</div>
<p>and replacing <span class="math notranslate nohighlight">\(1/\tau\)</span> with <span class="math notranslate nohighlight">\(\lambda\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
\]</div>
<p>which is our Lasso cost function!</p>
<p>Plotting these prior functions shows us that we can use the parameter
<span class="math notranslate nohighlight">\(\lambda\)</span> to shrink or increase the role of a given parameter
<span class="math notranslate nohighlight">\(\beta_j\)</span>. The variance for the Laplace distribution is
<span class="math notranslate nohighlight">\(2\tau^2=1/\lambda\)</span> while for the Gaussian distribution it is
<span class="math notranslate nohighlight">\(\sigma^2=1/(2\lambda)\)</span>. Thus, increasing the variance means
decreasing <span class="math notranslate nohighlight">\(\lambda\)</span> and shrinking the variance means increasing
<span class="math notranslate nohighlight">\(\lambda\)</span>. When we increase <span class="math notranslate nohighlight">\(\lambda\)</span>, this corresponds to shrinking the role of less important features (small singular values).</p>
</div>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
<a class='left-prev' id="prev-link" href="chapter1.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title"><span class="section-number">3. </span>Linear Regression</p>
</div>
</a>
<a class='right-next' id="next-link" href="chapter3.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title"><span class="section-number">5. </span>Resampling Methods</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
</div>
</div>
</div>
<footer class="footer">
<p>
By Morten Hjorth-Jensen<br/>
&copy; Copyright 2021.<br/>
</p>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.be7d3bbb2ef33a8344ce.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink