lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2022-10-05 07:55:07 +02:00
parent b00518e915
commit 031d352181
31 changed files with 3581 additions and 2153 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter2.html
+79 -80
View File
@@ -551,7 +551,7 @@ later shrinkage methods like Ridge and Lasso regressions.</p>
<p>This is given by the <strong>Singular Value Decomposition</strong> (SVD) algorithm,
perhaps the most powerful linear algebra algorithm. The SVD provides
a numerically stable matrix decomposition that is used in a large
swath oc applications and the decomposition is always stable
swath of applications and the decomposition is always stable
numerically.</p>
<p>In machine learning it plays a central role in dealing with for
example design matrices that may be near singular or singular.
@@ -565,7 +565,7 @@ when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</s
are problems with near singular or singular matrices. The column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>
may be linearly dependent, normally referred to as super-collinearity.<br />
This means that the matrix may be rank deficient and it is basically impossible to
to model the data using linear regression. As an example, consider the matrix</p>
model the data using linear regression. As an example, consider the matrix</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
@@ -585,7 +585,7 @@ to model the data using linear regression. As an example, consider the matrix</p
the first column is the row-wise sum of the other two columns. The rank (more correct,
the column rank) of a matrix is the dimension of the space spanned by the
column vectors. Hence, the rank of <span class="math notranslate nohighlight">\(\mathbf{X}\)</span> is equal to the number
of linearly independent columns. In this particular case the matrix has rank 2.</p>
of linearly independent columns. In this particular case the matrix has rank 1.</p>
<p>Super-collinearity of an <span class="math notranslate nohighlight">\((n \times p)\)</span>-dimensional design matrix <span class="math notranslate nohighlight">\(\mathbf{X}\)</span> implies
that the inverse of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this</p>
<div class="math notranslate nohighlight">
@@ -613,7 +613,7 @@ This is equivalent to saying that the matrix <span class="math notranslate nohig
\]</div>
<p>has linearly dependent column vectors, we will not be able to compute the inverse
of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> and we cannot find the parameters (estimators) <span class="math notranslate nohighlight">\(\beta_i\)</span>.
The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})^{-1}\)</span> exits.
The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})\)</span> can be inverted.
This is more likely to happen when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is high-dimensional. In this case it is likely to encounter a situation where
the regression parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> cannot be estimated.</p>
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
@@ -625,7 +625,7 @@ the regression parameters <span class="math notranslate nohighlight">\(\beta_i\)
</div>
<div class="section" id="basic-math-of-the-svd">
<h2><span class="section-number">4.3. </span>Basic math of the SVD<a class="headerlink" href="#basic-math-of-the-svd" title="Permalink to this headline"></a></h2>
<p>From standard linear algebra we know that a square matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> can be diagonalized if and only it is
<p>From standard linear algebra we know that a square matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> can be diagonalized if and only if it is
a so-called <a class="reference external" href="https://en.wikipedia.org/wiki/Normal_matrix">normal matrix</a>, that is if <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times n}\)</span>
we have <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X}\)</span> or if <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{C}}^{n\times n}\)</span> we have <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X}\)</span>.
The matrix has then a set of eigenpairs</p>
@@ -813,7 +813,6 @@ The simple answer is to use the linear algebra function for the pseudoinverse, t
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">V</span><span class="p">,</span><span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">invD</span><span class="p">,</span><span class="n">UT</span><span class="p">))</span>
<span class="c1">#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])</span>
<span class="c1"># Non-singular square matrix</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span> <span class="p">[</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">],[</span><span class="mi">2</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">],[</span><span class="mi">3</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">]])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
@@ -849,7 +848,7 @@ test VT
rectangular matrices where the number of rows and columns are not equal.</p>
<p>It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.</p>
<p>Using the SVD we can obtain the pseudoinverse of a matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> (labeled here as <span class="math notranslate nohighlight">\(\boldsymbol{A}_{\mathrm{PI}}\)</span></p>
<p>Using the SVD we can obtain the pseudoinverse (PI) of a matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> (labeled here as <span class="math notranslate nohighlight">\(\boldsymbol{A}_{\mathrm{PI}}\)</span></p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T,
@@ -919,7 +918,7 @@ x_{n-1,0} &amp; x_{n-1,1} &amp; x_{n-1,2}&amp; \dots &amp; \dots x_{n-1,p-1}\\
\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> is an orthogonal matrix of dimension <span class="math notranslate nohighlight">\(n\times n\)</span>, meaning that <span class="math notranslate nohighlight">\(\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n\)</span>. Here <span class="math notranslate nohighlight">\(\boldsymbol{I}_n\)</span> is the unit matrix of dimension <span class="math notranslate nohighlight">\(n \times n\)</span>.</p>
<p>Similarly, <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> is an orthogonal matrix of dimension <span class="math notranslate nohighlight">\(p\times p\)</span>, meaning that <span class="math notranslate nohighlight">\(\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p\)</span>. Here <span class="math notranslate nohighlight">\(\boldsymbol{I}_n\)</span> is the unit matrix of dimension <span class="math notranslate nohighlight">\(p \times p\)</span>.</p>
<p>Similarly, <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span> is an orthogonal matrix of dimension <span class="math notranslate nohighlight">\(p\times p\)</span>, meaning that <span class="math notranslate nohighlight">\(\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p\)</span>. Here <span class="math notranslate nohighlight">\(\boldsymbol{I}_p\)</span> is the unit matrix of dimension <span class="math notranslate nohighlight">\(p \times p\)</span>.</p>
<p>Finally <span class="math notranslate nohighlight">\(\boldsymbol{\Sigma}\)</span> contains the singular values <span class="math notranslate nohighlight">\(\sigma_i\)</span>. This matrix has dimension <span class="math notranslate nohighlight">\(n\times p\)</span> and the singular values <span class="math notranslate nohighlight">\(\sigma_i\)</span> are all positive. The non-zero values are ordered in descending order, that is</p>
<div class="math notranslate nohighlight">
\[
@@ -1057,7 +1056,7 @@ function, that is we have</p>
\[
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\]</div>
<p>This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).</p>
<p>This quantity defines what is called the Hessian matrix (the second derivative of the cost function we want to optimize).</p>
<p>The Hessian matrix plays an important role and is defined in this course as</p>
<div class="math notranslate nohighlight">
\[
@@ -1148,7 +1147,7 @@ We can rewrite the design/feature matrix in terms of its column vectors as</p>
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} &amp; x_{1,i} &amp; x_{2,i}&amp; \dots &amp; \dots x_{n-1,i}\end{bmatrix}.
\]</div>
<p>With these definitions, we can now rewrite our <span class="math notranslate nohighlight">\(2\times 2\)</span>
correlation/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a more general design/feature
matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>. This leads to a <span class="math notranslate nohighlight">\(p\times p\)</span>
covariance matrix for the vectors <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> with <span class="math notranslate nohighlight">\(i=0,1,\dots,p-1\)</span></p>
<div class="math notranslate nohighlight">
@@ -1207,10 +1206,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.04413933503955871
4.12330280229368
[[0.80162359 2.38222896]
[2.38222896 8.12167821]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.08873443359350565
3.7851533175757255
[[ 0.98248312 3.05483267]
[ 3.05483267 10.24784064]]
</pre></div>
</div>
</div>
@@ -1247,10 +1246,10 @@ a more brute force way. Here we scale the mean values for each column of the des
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.06786925114666595
1.9635449873404844
[[1. 0.65522261]
[0.65522261 1. ]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07858099596662704
2.071920625289855
[[1. 0.71822416]
[0.71822416 1. ]]
</pre></div>
</div>
</div>
@@ -1259,7 +1258,7 @@ a more brute force way. Here we scale the mean values for each column of the des
should be and that the matrix is symmetric. Furthermore, diagonalizing
this matrix we easily see that it is a positive definite matrix.</p>
<p>The above procedure with <strong>numpy</strong> can be made more compact if we use <strong>pandas</strong>.</p>
<p>We whow here how we can set up the correlation matrix using <strong>pandas</strong>, as done in this simple code</p>
<p>We know here how we can set up the correlation matrix using <strong>pandas</strong>, as done in this simple code</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
@@ -1280,30 +1279,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.27091656 -1.29083183]
[ 0.31980301 0.87495119]
[-0.10835935 1.61413333]
[ 0.5188328 2.80380438]
[-0.04996008 -1.95742107]
[ 1.19432526 2.68719389]
[ 0.19710439 1.35590603]
[-0.23857423 -2.50104946]
[-0.94054854 -2.09034902]
[-0.62170669 -1.49633743]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ -2.84861838 -10.07337358]
[ 0.53938383 2.59445979]
[ -0.40980089 -0.48871288]
[ 0.05834332 -0.39384255]
[ 2.25385387 7.58112299]
[ 0.68246434 2.46650488]
[ -0.25366775 -1.97047717]
[ 0.79081838 2.03807267]
[ -0.06150169 -0.57109235]
[ -0.75127504 -1.18266178]]
0 1
0 -2.848618 -10.073374
1 0.539384 2.594460
2 -0.409801 -0.488713
3 0.058343 -0.393843
4 2.253854 7.581123
5 0.682464 2.466505
6 -0.253668 -1.970477
7 0.790818 2.038073
8 -0.061502 -0.571092
9 -0.751275 -1.182662
0 1
0 -0.270917 -1.290832
1 0.319803 0.874951
2 -0.108359 1.614133
3 0.518833 2.803804
4 -0.049960 -1.957421
5 1.194325 2.687194
6 0.197104 1.355906
7 -0.238574 -2.501049
8 -0.940549 -2.090349
9 -0.621707 -1.496337
0 1
0 1.000000 0.800615
1 0.800615 1.000000
0 1.000000 0.984525
1 0.984525 1.000000
</pre></div>
</div>
</div>
@@ -1360,37 +1359,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.0 0.084006 0.079882 0.084682 0.084092 0.083417 0.076315 0.076097
2 0.0 0.079882 0.077644 0.078542 0.078962 0.079424 0.069534 0.069977
3 0.0 0.084682 0.078542 0.090649 0.088758 0.086665 0.085105 0.084008
4 0.0 0.084092 0.078962 0.088758 0.087573 0.086250 0.082424 0.081832
5 0.0 0.083417 0.079424 0.086665 0.086250 0.085776 0.079486 0.079438
6 0.0 0.076315 0.069534 0.085105 0.082424 0.079486 0.082288 0.080588
7 0.0 0.076097 0.069977 0.084008 0.081832 0.079438 0.080588 0.079264
8 0.0 0.076022 0.070618 0.082990 0.081357 0.079553 0.078908 0.077986
9 0.0 0.076079 0.071460 0.082027 0.080984 0.079823 0.077219 0.076729
10 0.0 0.068075 0.061188 0.078143 0.075043 0.071666 0.077200 0.075149
11 0.0 0.067712 0.061308 0.077144 0.074420 0.071445 0.075770 0.074006
12 0.0 0.067499 0.061604 0.076264 0.073938 0.071388 0.074418 0.072955
13 0.0 0.067443 0.062089 0.075498 0.073597 0.071505 0.073134 0.071991
14 0.0 0.067547 0.062777 0.074845 0.073400 0.071804 0.071908 0.071106
1 0.0 0.090241 0.082140 0.090564 0.084086 0.078082 0.082282 0.076619
2 0.0 0.082140 0.075227 0.083102 0.077428 0.072150 0.075982 0.070945
3 0.0 0.090564 0.083102 0.096893 0.090268 0.084107 0.091647 0.085571
4 0.0 0.084086 0.077428 0.090268 0.084286 0.078707 0.085655 0.080120
5 0.0 0.078082 0.072150 0.084107 0.078707 0.073657 0.080061 0.075020
6 0.0 0.082282 0.075982 0.091647 0.085655 0.080061 0.089082 0.083380
7 0.0 0.076619 0.070945 0.085571 0.080120 0.075020 0.083380 0.078158
8 0.0 0.071394 0.066284 0.079944 0.074984 0.070333 0.078082 0.073299
9 0.0 0.066569 0.061966 0.074729 0.070216 0.065973 0.073159 0.068776
10 0.0 0.073831 0.068541 0.084523 0.079224 0.074258 0.083779 0.078587
11 0.0 0.068867 0.064081 0.079021 0.074183 0.069640 0.078484 0.073716
12 0.0 0.064284 0.059952 0.073925 0.069506 0.065349 0.073567 0.069187
13 0.0 0.060048 0.056127 0.069202 0.065165 0.061359 0.068999 0.064974
14 0.0 0.056131 0.052581 0.064823 0.061133 0.057648 0.064753 0.061054
8 9 10 11 12 13 14
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.076022 0.076079 0.068075 0.067712 0.067499 0.067443 0.067547
2 0.070618 0.071460 0.061188 0.061308 0.061604 0.062089 0.062777
3 0.082990 0.082027 0.078143 0.077144 0.076264 0.075498 0.074845
4 0.081357 0.080984 0.075043 0.074420 0.073938 0.073597 0.073400
5 0.079553 0.079823 0.071666 0.071445 0.071388 0.071505 0.071804
6 0.078908 0.077219 0.077200 0.075770 0.074418 0.073134 0.071908
7 0.077986 0.076729 0.075149 0.074006 0.072955 0.071991 0.071106
8 0.077140 0.076349 0.073080 0.072240 0.071510 0.070887 0.070370
9 0.076349 0.076066 0.070961 0.070443 0.070056 0.069801 0.069681
10 0.073080 0.070961 0.073601 0.071922 0.070288 0.068689 0.067110
11 0.072240 0.070443 0.071922 0.070466 0.069065 0.067709 0.066388
12 0.071510 0.070056 0.070288 0.069065 0.067907 0.066808 0.065761
13 0.070887 0.069801 0.068689 0.067709 0.066808 0.065983 0.065228
14 0.070370 0.069681 0.067110 0.066388 0.065761 0.065228 0.064787
1 0.071394 0.066569 0.073831 0.068867 0.064284 0.060048 0.056131
2 0.066284 0.061966 0.068541 0.064081 0.059952 0.056127 0.052581
3 0.079944 0.074729 0.084523 0.079021 0.073925 0.069202 0.064823
4 0.074984 0.070216 0.079224 0.074183 0.069506 0.065165 0.061133
5 0.070333 0.065973 0.074258 0.069640 0.065349 0.061359 0.057648
6 0.078082 0.073159 0.083779 0.078484 0.073567 0.068999 0.064753
7 0.073299 0.068776 0.078587 0.073716 0.069187 0.064974 0.061054
8 0.068841 0.064684 0.073750 0.069268 0.065095 0.061209 0.057588
9 0.064684 0.060863 0.069242 0.065118 0.061272 0.057686 0.054340
10 0.073750 0.069242 0.079948 0.075028 0.070450 0.066189 0.062220
11 0.069268 0.065118 0.075028 0.070494 0.066270 0.062333 0.058663
12 0.065095 0.061272 0.070450 0.066270 0.062370 0.058732 0.055337
13 0.061209 0.057686 0.066189 0.062333 0.058732 0.055369 0.052227
14 0.057588 0.054340 0.062220 0.058663 0.055337 0.052227 0.049318
</pre></div>
</div>
</div>
@@ -1668,7 +1667,7 @@ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\bolds
<p>This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a class="reference external" href="https://cvxopt.org/">CVXOPT</a>. We will discuss this later.</p>
<p>Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
diagonal. In this case we have an equal number of rows and columns <span class="math notranslate nohighlight">\(n=p\)</span>.</p>
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\beta}\)</span> and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\beta}\)</span> and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
@@ -1706,7 +1705,7 @@ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\ve
0 &amp;\mathrm{if} &amp; \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
\end{split}\]</div>
<p>Plotting these results (<a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf">figure in handwritten notes for week 36</a>) shows clearly that Lasso regression suppresses (sets to zero) values of <span class="math notranslate nohighlight">\(\beta_i\)</span> for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>. Ridge regression reduces on the other hand the values of <span class="math notranslate nohighlight">\(\beta_i\)</span> as function of <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
<p>As another examples,
<p>As another example,
let us assume we have a data set with outputs/targets given by the vector</p>
<div class="math notranslate nohighlight">
\[\begin{split}
@@ -1878,12 +1877,12 @@ Training MSE for OLS
<img alt="_images/chapter2_252_1.png" src="_images/chapter2_252_1.png" />
</div>
</div>
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
<p>This happens also for Lasso regression, as seen from the next code
output. The difference is that Lasso shrinks the values of <span class="math notranslate nohighlight">\(\beta\)</span> to
zero at a much earlier stage and the results flatten out. We see that
Lasso gives also an excellent fit for small values of <span class="math notranslate nohighlight">\(\lambda\)</span> and
shows rthe best performance of the three regression methods.</p>
shows the best performance of the three regression methods.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
@@ -2426,7 +2425,7 @@ We define this distribution as</p>
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
\]</div>
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<p>Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
<p>Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
@@ -2497,7 +2496,7 @@ p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
<p>The product rule (aka joint probability) is given by</p>
<div class="math notranslate nohighlight">
\[
p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
p(X \cap Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
\]</div>
<p>where we read <span class="math notranslate nohighlight">\(p(X\vert Y)\)</span> as the likelihood of obtaining <span class="math notranslate nohighlight">\(X\)</span> given <span class="math notranslate nohighlight">\(Y\)</span>.</p>
<p>If we have independent events then <span class="math notranslate nohighlight">\(p(X,Y)=p(X)p(Y)\)</span>.</p>
@@ -2973,7 +2972,7 @@ parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> as func
noise. Here we recommend to use <span class="math notranslate nohighlight">\(\sigma^2=1\)</span> as variance for the
added noise (which follows a normal distribution with mean value zero).
Comment your results. If you have a large noise term, do the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> vary more as function
model complexity? And what about their variance?</p>
of model complexity? And what about their variance?</p>
</div>
<div class="section" id="linking-bayes-theorem-with-ridge-and-lasso-regression">
<h2><span class="section-number">4.14. </span>Linking Bayes’ Theorem with Ridge and Lasso Regression<a class="headerlink" href="#linking-bayes-theorem-with-ridge-and-lasso-regression" title="Permalink to this headline"></a></h2>
@@ -3002,7 +3001,7 @@ p(\boldsymbol{\beta}\vert\boldsymbol{D}).
\[
p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}).
\]</div>
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\beta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\beta}\)</span>!</p>
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\beta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\beta})\)</span>!</p>
<p>With the posterior probability defined by a likelihood which we have
already modeled and an unknown prior, we are now ready to make
additional models for the prior.</p>
@@ -3047,12 +3046,12 @@ logarithm of the posterior probability and leaving out the
constants terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
\]</div>
<p>and replacing <span class="math notranslate nohighlight">\(1/\tau\)</span> with <span class="math notranslate nohighlight">\(\lambda\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
\]</div>
<p>which is our Lasso cost function!</p>
<p>Plotting these prior functions shows us that we can use the parameter