lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 14:05:16 +02:00
parent 30d664a82b
commit ad92592adb
76 changed files with 5646 additions and 2878 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter2.html
+137 -104
View File
@@ -159,7 +159,7 @@
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
7. Support Vector Machines, overarching aims
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
@@ -171,12 +171,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
8. Decision trees, overarching aims
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
@@ -188,12 +188,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
10. Basic ideas of the Principal Component Analysis (PCA)
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="Clustering.html">
11. Clustering Analysis
12. Clustering Analysis
</a>
</li>
</ul>
@@ -205,12 +205,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
12. Neural networks
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
13. Building a Feed Forward Neural Network
14. Building a Feed Forward Neural Network
</a>
</li>
</ul>
@@ -349,6 +349,11 @@
4.13. Bayes’ Theorem and Ridge and Lasso Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#linking-bayes-theorem-with-ridge-and-lasso-regression">
4.14. Linking Bayes’ Theorem with Ridge and Lasso Regression
</a>
</li>
</ul>
</nav>
@@ -365,10 +370,10 @@
<div class="section" id="mathematical-interpretation-of-ordinary-least-squares">
<h2><span class="section-number">4.1. </span>Mathematical Interpretation of Ordinary Least Squares<a class="headerlink" href="#mathematical-interpretation-of-ordinary-least-squares" title="Permalink to this headline"></a></h2>
<p>What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).</p>
<p>We have shown that in ordinary least squares the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> are given by</p>
<p>We have shown that in ordinary least squares (OLS) the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> are given by</p>
<div class="math notranslate nohighlight">
\[
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\]</div>
<p>The <strong>hat</strong> over <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> means we have the optimal parameters after minimization of the cost function.</p>
<p>This means that our best model is defined as</p>
@@ -386,7 +391,7 @@
\[
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
\]</div>
<p>The matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> has the important property that <span class="math notranslate nohighlight">\(\boldsymbol{A}^2=\boldsymbol{A}\)</span>. This is the definition of a projection matrix.
<p>The matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> has the important property that <span class="math notranslate nohighlight">\(\boldsymbol{A}^2=\boldsymbol{A}\)</span>. This is the definition of a <a class="reference external" href="https://en.wikipedia.org/wiki/Projection_matrix">projection matrix</a>.
We can then interpret our optimal model <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> as being represented by an orthogonal projection of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> onto a space defined by the column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. In our case here the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.</p>
<p>We have defined the residual error as</p>
<div class="math notranslate nohighlight">
@@ -659,7 +664,7 @@ example</p>
</div>
<div class="section" id="code-for-svd-and-inversion-of-matrices">
<h2><span class="section-number">4.5. </span>Code for SVD and Inversion of Matrices<a class="headerlink" href="#code-for-svd-and-inversion-of-matrices" title="Permalink to this headline"></a></h2>
<p>How do we use the SVD to invert a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^\boldsymbol{X}\)</span> which is singular or near singular?
<p>How do we use the SVD to invert a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> which is singular or near singular?
The simple answer is to use the linear algebra function for the pseudoinverse, that is</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -1096,10 +1101,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.059680303635344434
4.260227742627244
[[0.89704131 2.55795935]
[2.55795935 8.24340751]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.1831277634928002
3.3041320306136366
[[1.03929932 3.07670437]
[3.07670437 9.7701384 ]]
</pre></div>
</div>
</div>
@@ -1136,10 +1141,10 @@ a more brute force way. Here we scale the mean values for each column of the des
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08362077210702115
1.8429949116841184
[[1. 0.6996584]
[0.6996584 1. ]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.09456011349477329
1.6600855222624895
[[1. 0.70183798]
[0.70183798 1. ]]
</pre></div>
</div>
</div>
@@ -1169,30 +1174,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-1.25981533 -3.84976606]
[-0.52570079 -0.62262506]
[ 0.51707172 3.46973015]
[-0.47265243 -1.87030083]
[ 0.73734906 2.46183815]
[ 0.4918372 2.05531364]
[ 0.3893239 0.47862383]
[-0.41237437 -0.85058354]
[ 0.77622336 0.54697204]
[-0.24126232 -1.81920231]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 0.54316925 0.40968888]
[-1.8393131 -5.8788431 ]
[ 0.24613822 0.9957744 ]
[ 1.38892672 3.93679587]
[ 0.06540809 -0.84008474]
[ 0.49841285 2.31081134]
[-0.49323032 -1.25263916]
[ 0.08380813 -0.7925146 ]
[ 0.64078247 2.92814088]
[-1.13410232 -1.81712976]]
0 1
0 -1.259815 -3.849766
1 -0.525701 -0.622625
2 0.517072 3.469730
3 -0.472652 -1.870301
4 0.737349 2.461838
5 0.491837 2.055314
6 0.389324 0.478624
7 -0.412374 -0.850584
8 0.776223 0.546972
9 -0.241262 -1.819202
0 0.543169 0.409689
1 -1.839313 -5.878843
2 0.246138 0.995774
3 1.388927 3.936796
4 0.065408 -0.840085
5 0.498413 2.310811
6 -0.493230 -1.252639
7 0.083808 -0.792515
8 0.640782 2.928141
9 -1.134102 -1.817130
0 1
0 1.000000 0.881323
1 0.881323 1.000000
0 1.000000 0.942726
1 0.942726 1.000000
</pre></div>
</div>
</div>
@@ -1249,37 +1254,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.0 0.082272 0.078892 0.081102 0.080738 0.080182 0.072012 0.072246
2 0.0 0.078892 0.076822 0.075975 0.076166 0.076251 0.066705 0.067202
3 0.0 0.081102 0.075975 0.085259 0.084103 0.082616 0.078750 0.078649
4 0.0 0.080738 0.076166 0.084103 0.083246 0.082091 0.077338 0.077407
5 0.0 0.080182 0.076251 0.082616 0.082091 0.081309 0.075567 0.075818
6 0.0 0.072012 0.066705 0.078750 0.077338 0.075567 0.074693 0.074434
7 0.0 0.072246 0.067202 0.078649 0.077407 0.075818 0.074434 0.074285
8 0.0 0.072468 0.067718 0.078474 0.077416 0.076026 0.074077 0.074044
9 0.0 0.072661 0.068244 0.078197 0.077339 0.076169 0.073591 0.073681
10 0.0 0.063190 0.058247 0.070949 0.069542 0.067794 0.068593 0.068293
11 0.0 0.063523 0.058719 0.071156 0.069853 0.068213 0.068717 0.068491
12 0.0 0.063880 0.059225 0.071359 0.070166 0.068640 0.068821 0.068673
13 0.0 0.064257 0.059765 0.071549 0.070473 0.069071 0.068896 0.068828
14 0.0 0.064647 0.060335 0.071714 0.070764 0.069496 0.068928 0.068945
1 0.0 0.080633 0.085811 0.085748 0.084826 0.083221 0.080341 0.078342
2 0.0 0.085811 0.092659 0.093248 0.092932 0.091659 0.088414 0.086561
3 0.0 0.085748 0.093248 0.096639 0.096802 0.095941 0.093928 0.092288
4 0.0 0.084826 0.092932 0.096802 0.097346 0.096763 0.094774 0.093328
5 0.0 0.083221 0.091659 0.095941 0.096763 0.096406 0.094513 0.093236
6 0.0 0.080341 0.088414 0.093928 0.094774 0.094513 0.093573 0.092375
7 0.0 0.078342 0.086561 0.092288 0.093328 0.093236 0.092375 0.091318
8 0.0 0.076213 0.084461 0.090355 0.091535 0.091578 0.090817 0.089881
9 0.0 0.074037 0.082239 0.088258 0.089540 0.089693 0.089039 0.088209
10 0.0 0.073546 0.081474 0.088137 0.089326 0.089437 0.089352 0.088484
11 0.0 0.071453 0.079337 0.086052 0.087334 0.087544 0.087526 0.086757
12 0.0 0.069371 0.077163 0.083905 0.085253 0.085544 0.085598 0.084916
13 0.0 0.067326 0.074996 0.081744 0.083138 0.083495 0.083622 0.083015
14 0.0 0.065334 0.072862 0.079597 0.081024 0.081435 0.081632 0.081092
8 9 10 11 12 13 14
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.072468 0.072661 0.063190 0.063523 0.063880 0.064257 0.064647
2 0.067718 0.068244 0.058247 0.058719 0.059225 0.059765 0.060335
3 0.078474 0.078197 0.070949 0.071156 0.071359 0.071549 0.071714
4 0.077416 0.077339 0.069542 0.069853 0.070166 0.070473 0.070764
5 0.076026 0.076169 0.067794 0.068213 0.068640 0.069071 0.069496
6 0.074077 0.073591 0.068593 0.068717 0.068821 0.068896 0.068928
7 0.074044 0.073681 0.068293 0.068491 0.068673 0.068828 0.068945
8 0.073926 0.073695 0.067889 0.068165 0.068427 0.068667 0.068873
9 0.073695 0.073607 0.067355 0.067711 0.068056 0.068385 0.068686
10 0.067889 0.067355 0.063925 0.064013 0.064075 0.064101 0.064078
11 0.068165 0.067711 0.064013 0.064156 0.064273 0.064357 0.064393
12 0.068427 0.068056 0.064075 0.064273 0.064449 0.064591 0.064689
13 0.068667 0.068385 0.064101 0.064357 0.064591 0.064795 0.064958
14 0.068873 0.068686 0.064078 0.064393 0.064689 0.064958 0.065188
1 0.076213 0.074037 0.073546 0.071453 0.069371 0.067326 0.065334
2 0.084461 0.082239 0.081474 0.079337 0.077163 0.074996 0.072862
3 0.090355 0.088258 0.088137 0.086052 0.083905 0.081744 0.079597
4 0.091535 0.089540 0.089326 0.087334 0.085253 0.083138 0.081024
5 0.091578 0.089693 0.089437 0.087544 0.085544 0.083495 0.081435
6 0.090817 0.089039 0.089352 0.087526 0.085598 0.083622 0.081632
7 0.089881 0.088209 0.088484 0.086757 0.084916 0.083015 0.081092
8 0.088556 0.086985 0.087242 0.085610 0.083855 0.082032 0.080182
9 0.086985 0.085510 0.085763 0.084221 0.082549 0.080806 0.079029
10 0.087242 0.085763 0.086430 0.084864 0.083177 0.081425 0.079643
11 0.085610 0.084221 0.084864 0.083384 0.081778 0.080101 0.078390
12 0.083855 0.082549 0.083177 0.081778 0.080249 0.078646 0.077005
13 0.082032 0.080806 0.081425 0.080101 0.078646 0.077115 0.075542
14 0.080182 0.079029 0.079643 0.078390 0.077005 0.075542 0.074036
</pre></div>
</div>
</div>
@@ -1758,13 +1763,11 @@ We select values of the hyperparameter <span class="math notranslate nohighlight
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[2. 2.]
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training MSE for OLS
Training MSE for OLS
3.0
</pre></div>
</div>
<img alt="_images/chapter2_245_2.png" src="_images/chapter2_245_2.png" />
<img alt="_images/chapter2_245_1.png" src="_images/chapter2_245_1.png" />
</div>
</div>
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
@@ -2438,6 +2441,7 @@ Optimal solution found.
</div>
</div>
</div>
<p><strong>More text will be added to this example.</strong></p>
</div>
<div class="section" id="linking-the-regression-analysis-with-a-statistical-interpretation">
<h2><span class="section-number">4.11. </span>Linking the regression analysis with a statistical interpretation<a class="headerlink" href="#linking-the-regression-analysis-with-a-statistical-interpretation" title="Permalink to this headline"></a></h2>
@@ -2619,7 +2623,9 @@ We can now rewrite the above probability as</p>
\[
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
\]</div>
<p>It is a conditional probability (see below) and reads as the likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<p>It is a conditional probability (see below) and reads as the
likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<p>In statistics, maximum likelihood estimation (MLE) is a method of
estimating the parameters of an assumed probability distribution,
given some observed data. This is achieved by maximizing a likelihood
@@ -2663,12 +2669,12 @@ Bayes’ theorem is extremely simple to derive. But to do so we need some basic
<p>Assume we have two domains of events <span class="math notranslate nohighlight">\(X=[x_0,x_1,\dots,x_{n-1}]\)</span> and <span class="math notranslate nohighlight">\(Y=[y_0,y_1,\dots,y_{n-1}]\)</span>.</p>
<p>We define also the likelihood for <span class="math notranslate nohighlight">\(X\)</span> and <span class="math notranslate nohighlight">\(Y\)</span> as <span class="math notranslate nohighlight">\(p(X)\)</span> and <span class="math notranslate nohighlight">\(p(Y)\)</span> respectively.
The likelihood of a specific event <span class="math notranslate nohighlight">\(x_i\)</span> (or <span class="math notranslate nohighlight">\(y_i\)</span>) is then written as <span class="math notranslate nohighlight">\(p(X=x_i)\)</span> or just <span class="math notranslate nohighlight">\(p(x_i)=p_i\)</span>.</p>
<p><strong>Union of events is given by.</strong></p>
<p>The union of events is given by</p>
<div class="math notranslate nohighlight">
\[
p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
\]</div>
<p><strong>The product rule (aka joint probability) is given by.</strong></p>
<p>The product rule (aka joint probability) is given by</p>
<div class="math notranslate nohighlight">
\[
p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
@@ -2700,37 +2706,68 @@ p(X\vert Y)= \frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}=\frac{p(Y\vert
evaluated for the observed data <span class="math notranslate nohighlight">\(Y\)</span> and can be viewed as a function of
the parameter space represented by <span class="math notranslate nohighlight">\(X\)</span>. This function is not
necesseraly normalized and is normally called the likelihood function.</p>
<p>The function <span class="math notranslate nohighlight">\(p(X)\)</span> on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution.</p>
<p>The function <span class="math notranslate nohighlight">\(p(X)\)</span> on the right hand side is called the prior while
the function on the left hand side is the called the posterior
probability. The denominator on the right hand side serves as a
normalization factor for the posterior distribution.</p>
<p>Let us try to illustrate Bayes’ theorem through an example.</p>
<p>Let us suppose that you are undergoing a series of mammography scans in
order to rule out possible breast cancer cases. We define the
<p>Let us suppose that you are undergoing a series of mammography scans
in order to rule out possible breast cancer cases. We define the
sensitivity for a positive event by the variable <span class="math notranslate nohighlight">\(X\)</span>. It takes binary
values with <span class="math notranslate nohighlight">\(X=1\)</span> representing a positive event and <span class="math notranslate nohighlight">\(X=0\)</span> being a
negative event. We reserve <span class="math notranslate nohighlight">\(Y\)</span> as a classification parameter for
either a negative or a positive breast cancer confirmation. (Short note on wordings: positive here means having breast cancer, although none of us would consider this being a positive thing).</p>
either a negative or a positive breast cancer confirmation. (Short
note on wordings: positive here means having breast cancer, although
none of us would consider this being a positive thing).</p>
<p>We let <span class="math notranslate nohighlight">\(Y=1\)</span> represent the the case of having breast cancer and <span class="math notranslate nohighlight">\(Y=0\)</span> as not.</p>
<p>Let us assume that if you have breast cancer, the test will be positive with a probability of <span class="math notranslate nohighlight">\(0.8\)</span>, that is we have</p>
<p>Let us assume that if you have breast cancer, the test will be positive with a probability of <span class="math notranslate nohighlight">\(0.8\)</span> (the numbers here are all made up),
that is we have</p>
<div class="math notranslate nohighlight">
\[
p(X=1\vert Y=1) =0.8.
\]</div>
<p>This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of <span class="math notranslate nohighlight">\(80\%\)</span> for having cancer.
It is however not correct, as the following Bayesian analysis shows.</p>
<p>If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number.
Let us assume that the prior probability in the population as a whole is</p>
<p>This obviously sounds scary since many would conclude that if the test
is positive, there is a likelihood of <span class="math notranslate nohighlight">\(80\%\)</span> for having cancer. It is
however not correct, as the following Bayesian analysis shows. The correct question to pose is <em>what is the probability of having breast cancer in case of a positive test?</em>
We are thus interested in</p>
<div class="math notranslate nohighlight">
\[
p(Y=1\vert X=1),
\]</div>
<p>instead of <span class="math notranslate nohighlight">\(p(X=1\vert Y=1)\)</span>.</p>
<p>If we look at various national surveys on breast cancer, the general
likelihood of developing breast cancer is a very small number. Let us
assume that the prior probability in the population as a whole is</p>
<div class="math notranslate nohighlight">
\[
p(Y=1) =0.004.
\]</div>
<p>We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have</p>
<p>We need also to account for the fact that the test may produce a false
positive result (false alarm). Let us here assume that we have</p>
<div class="math notranslate nohighlight">
\[
p(X=1\vert Y=0) =0.1.
\]</div>
<p>Using Bayes’ theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute</p>
<p>Using Bayes’ theorem we can then find the posterior probability that
the person has breast cancer in case of a positive test, that is we
can compute</p>
<!-- Equation labels as ordinary links -->
<div id="_auto2"></div>
<div class="math notranslate nohighlight">
\[
p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031.
\begin{equation}
p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=
\label{_auto2} \tag{2}
\end{equation}
\]</div>
<!-- Equation labels as ordinary links -->
<div id="_auto3"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}
\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031.
\label{_auto3} \tag{3}
\end{equation}
\]</div>
<p>That is, in case of a positive test, there is only a <span class="math notranslate nohighlight">\(3\%\)</span> chance of having breast cancer!</p>
</div>
@@ -2745,8 +2782,11 @@ more intuitive way of understanding what Ridge and Lasso express.</p>
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> using OLS, Ridge and Lasso regression. For
OLS, you will notice as function of the noise and polynomial degree,
that the parameters <span class="math notranslate nohighlight">\(\beta\)</span> will fluctuate from order to order in the
polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS.</p>
<p>For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one.</p>
polynomial fit and that for larger and larger polynomial degrees of
freedom, the parameters will tend to increase in value for OLS.</p>
<p>For Ridge and Lasso regression, the higher order parameters will
typically be reduced, providing thereby less fluctuations from one
order to another one.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
@@ -2802,8 +2842,8 @@ polynomial fit and that for larger and larger polynomial degrees of freedom, the
<span class="c1"># Compute the MSE and print it</span>
<span class="n">MSERidgePredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span>
<span class="n">MSELassoPredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictLasso</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">lmb</span><span class="p">,</span><span class="n">RegRidge</span><span class="o">.</span><span class="n">coef_</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">lmb</span><span class="p">,</span><span class="n">RegLasso</span><span class="o">.</span><span class="n">coef_</span><span class="p">)</span>
<span class="c1"># print(lmb,RegRidge.coef_)</span>
<span class="c1"># print(lmb,RegLasso.coef_)</span>
<span class="c1"># Now plot the results</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSERidgePredict</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE Ridge Test&#39;</span><span class="p">)</span>
@@ -2819,22 +2859,15 @@ polynomial fit and that for larger and larger polynomial degrees of freedom, the
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.0169643 0.27924636 -1.4087793 1.03308408 0. ]
Test MSE OLS
0.958228616652075
0.001 [ 1.0170259 0.27852549 -1.40702 1.03193199 0. ]
0.001 [ 1.034342 -0.18063928 -0. 0. 0. ]
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.021544346900318832 [ 1.01825571 0.26412372 -1.37186301 1.00890601 0. ]
0.021544346900318832 [ 0.92280994 -0. -0. -0. 0. ]
0.46415888336127775 [ 1.0344707 0.07160764 -0.89928965 0.69843037 0. ]
0.46415888336127775 [0.48019541 0. 0. 0. 0. ]
10.0 [ 1.04529095 -0.18224665 -0.15751596 0.16831012 0. ]
10.0 [0. 0. 0. 0. 0.]
</pre></div>
</div>
<img alt="_images/chapter2_313_2.png" src="_images/chapter2_313_2.png" />
<img alt="_images/chapter2_316_1.png" src="_images/chapter2_316_1.png" />
</div>
</div>
<p>How can we understand this?</p>
<p>How can we understand this? <strong>More text to be added</strong>.</p>
</div>
<div class="section" id="linking-bayes-theorem-with-ridge-and-lasso-regression">
<h2><span class="section-number">4.14. </span>Linking Bayes’ Theorem with Ridge and Lasso Regression<a class="headerlink" href="#linking-bayes-theorem-with-ridge-and-lasso-regression" title="Permalink to this headline"></a></h2>
<p>Using Bayes’ theorem we can gain a better intuition about Ridge and Lasso regression.</p>
<p>For ordinary least squares we postulated that the maximum likelihood for the doamin of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> (one-dimensional case)</p>
<div class="math notranslate nohighlight">