lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

quick update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-10 16:18:42 +02:00
parent b3f74565d8
commit 3e5a8a25ec
17 changed files with 760 additions and 182 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/BookChapters/chapter3.do.txt
+2 -2
View File
@@ -1257,9 +1257,9 @@ This equation does not lead to a nice analytical equation as in either Ridge reg
===== Code for SVD and Inversion of Matrices =====
How do we use the SVD to invert a matrix $\bm{X}^\bm{X}$ which is singular or near singular?
The simple answer is to use the linear algebra function for pseudoinvers, that is
The simple answer is to use the linear algebra function for the pseudoinverse, that is
!bc pycod
Ainv = np.linlag.pinv(A)
#Ainv = np.linlag.pinv(A)
!ec
Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD.
doc/LectureNotes/_build/.doctrees/chapter2.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/_images/chapter2_287_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.0 KiB

doc/LectureNotes/_build/html/_images/chapter2_289_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

doc/LectureNotes/_build/html/_images/chapter2_291_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

doc/LectureNotes/_build/html/_images/chapter2_351_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 10 KiB

doc/LectureNotes/_build/html/_sources/chapter2.ipynb
+2 -2
View File
@@ -2133,7 +2133,7 @@
"## Code for SVD and Inversion of Matrices\n",
"\n",
"How do we use the SVD to invert a matrix $\\boldsymbol{X}^\\boldsymbol{X}$ which is singular or near singular?\n",
"The simple answer is to use the linear algebra function for pseudoinvers, that is"
"The simple answer is to use the linear algebra function for the pseudoinverse, that is"
]
},
{
@@ -2145,7 +2145,7 @@
},
"outputs": [],
"source": [
"Ainv = np.linlag.pinv(A)"
"#Ainv = np.linlag.pinv(A)"
]
},
{
doc/LectureNotes/_build/html/chapter2.html
+332 -80
View File
@@ -982,10 +982,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.01210814993019007
3.8867467323038865
[[0.91417278 2.88046462]
[2.88046462 9.9860803 ]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.033037772005753835
3.7371165871823337
[[ 1.22443803 3.75195757]
[ 3.75195757 12.47441766]]
</pre></div>
</div>
</div>
@@ -1022,10 +1022,10 @@ a more brute force way. Here we scale the mean values for each column of the des
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07423848370736122
1.725394195434945
[[1. 0.71606852]
[0.71606852 1. ]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08665060086846632
1.6796265324852733
[[1. 0.6183694]
[0.6183694 1. ]]
</pre></div>
</div>
</div>
@@ -1055,33 +1055,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 0.52687171 0.97622676]
[-2.35038714 -6.09976319]
[-0.55707065 -1.51576807]
[-0.98314755 -2.54096582]
[-0.46556436 -0.43549028]
[ 3.28310983 9.01503674]
[ 0.76388013 2.67714918]
[-0.49722108 -2.64669382]
[ 1.20184113 3.80026635]
[-0.92231203 -3.22999784]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 0.68002363 0.95517094]
[-0.53545715 0.03652792]
[ 1.33886902 4.84251485]
[ 0.20375701 -0.16861772]
[-2.04455272 -5.89742056]
[ 1.17449733 2.3728892 ]
[ 0.03019554 0.07581375]
[ 0.60949193 1.95628168]
[ 0.0530533 1.07839924]
[-1.5098779 -5.2515593 ]]
0 1
0 0.526872 0.976227
1 -2.350387 -6.099763
2 -0.557071 -1.515768
3 -0.983148 -2.540966
4 -0.465564 -0.435490
5 3.283110 9.015037
6 0.763880 2.677149
7 -0.497221 -2.646694
8 1.201841 3.800266
9 -0.922312 -3.229998
0 0.680024 0.955171
1 -0.535457 0.036528
2 1.338869 4.842515
3 0.203757 -0.168618
4 -2.044553 -5.897421
5 1.174497 2.372889
6 0.030196 0.075814
7 0.609492 1.956282
8 0.053053 1.078399
9 -1.509878 -5.251559
0 1
0 1.000000 0.988663
1 0.988663 1.000000
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
0 1.000000 0.959247
1 0.959247 1.000000
</pre></div>
</div>
</div>
@@ -1138,37 +1135,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.0 0.075212 0.075261 0.077160 0.075772 0.074134 0.069902 0.068276
2 0.0 0.075261 0.077533 0.076062 0.075562 0.074848 0.067985 0.066788
3 0.0 0.077160 0.076062 0.084092 0.081886 0.079365 0.078927 0.076730
4 0.0 0.075772 0.075562 0.081886 0.080115 0.078049 0.076398 0.074453
5 0.0 0.074134 0.074848 0.079365 0.078049 0.076469 0.073567 0.071887
6 0.0 0.069902 0.067985 0.078927 0.076398 0.073567 0.075821 0.073483
7 0.0 0.068276 0.066788 0.076730 0.074453 0.071887 0.073483 0.071313
8 0.0 0.066629 0.065591 0.074488 0.072471 0.070182 0.071096 0.069098
9 0.0 0.064964 0.064410 0.072198 0.070456 0.068460 0.068654 0.066835
10 0.0 0.061842 0.059571 0.071481 0.068920 0.066092 0.069814 0.067531
11 0.0 0.060308 0.058282 0.069516 0.067123 0.064472 0.067777 0.065618
12 0.0 0.058805 0.057035 0.067577 0.065355 0.062885 0.065762 0.063727
13 0.0 0.057332 0.055831 0.065660 0.063615 0.061330 0.063763 0.061855
14 0.0 0.055887 0.054674 0.063759 0.061896 0.059806 0.061774 0.059994
1 0.0 0.073902 0.077303 0.072279 0.077386 0.082866 0.063526 0.068073
2 0.0 0.077303 0.081480 0.074665 0.080336 0.086465 0.064929 0.069872
3 0.0 0.072279 0.074665 0.075754 0.080381 0.085289 0.069635 0.074099
4 0.0 0.077386 0.080336 0.080381 0.085597 0.091159 0.073333 0.078279
5 0.0 0.082866 0.086465 0.085289 0.091159 0.097452 0.077212 0.082687
6 0.0 0.063526 0.064929 0.069635 0.073333 0.077212 0.066047 0.069859
7 0.0 0.068073 0.069872 0.074099 0.078279 0.082687 0.069859 0.074096
8 0.0 0.073035 0.075289 0.078937 0.083657 0.088660 0.073966 0.078674
9 0.0 0.078451 0.081227 0.084181 0.089504 0.095173 0.078388 0.083618
10 0.0 0.055085 0.055774 0.062327 0.065204 0.068185 0.060520 0.063670
11 0.0 0.058892 0.059865 0.066249 0.069510 0.072910 0.064000 0.067504
12 0.0 0.063055 0.064354 0.070517 0.074207 0.078076 0.067769 0.071667
13 0.0 0.067610 0.069282 0.075163 0.079333 0.083728 0.071855 0.076189
14 0.0 0.072598 0.074696 0.080223 0.084931 0.089916 0.076283 0.081103
8 9 10 11 12 13 14
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.066629 0.064964 0.061842 0.060308 0.058805 0.057332 0.055887
2 0.065591 0.064410 0.059571 0.058282 0.057035 0.055831 0.054674
3 0.074488 0.072198 0.071481 0.069516 0.067577 0.065660 0.063759
4 0.072471 0.070456 0.068920 0.067123 0.065355 0.063615 0.061896
5 0.070182 0.068460 0.066092 0.064472 0.062885 0.061330 0.059806
6 0.071096 0.068654 0.069814 0.067777 0.065762 0.063763 0.061774
7 0.069098 0.066835 0.067531 0.065618 0.063727 0.061855 0.059994
8 0.067061 0.064982 0.065202 0.063415 0.061652 0.059908 0.058178
9 0.064982 0.063097 0.062822 0.061164 0.059531 0.057919 0.056326
10 0.065202 0.062822 0.065086 0.063122 0.061176 0.059245 0.057320
11 0.063415 0.061164 0.063122 0.061255 0.059406 0.057571 0.055744
12 0.061652 0.059531 0.061176 0.059406 0.057654 0.055916 0.054187
13 0.059908 0.057919 0.059245 0.057571 0.055916 0.054276 0.052645
14 0.058178 0.056326 0.057320 0.055744 0.054187 0.052645 0.051115
1 0.073035 0.078451 0.055085 0.058892 0.063055 0.067610 0.072598
2 0.075289 0.081227 0.055774 0.059865 0.064354 0.069282 0.074696
3 0.078937 0.084181 0.062327 0.066249 0.070517 0.075163 0.080223
4 0.083657 0.089504 0.065204 0.069510 0.074207 0.079333 0.084931
5 0.088660 0.095173 0.068185 0.072910 0.078076 0.083728 0.089916
6 0.073966 0.078388 0.060520 0.064000 0.067769 0.071855 0.076283
7 0.078674 0.083618 0.063670 0.067504 0.071667 0.076189 0.081103
8 0.083775 0.089300 0.067043 0.071268 0.075865 0.080871 0.086325
9 0.089300 0.095472 0.070653 0.075309 0.080387 0.085928 0.091979
10 0.067043 0.070653 0.056484 0.059455 0.062659 0.066116 0.069848
11 0.071268 0.075309 0.059455 0.062731 0.066272 0.070103 0.074248
12 0.075865 0.080387 0.062659 0.066272 0.070188 0.074432 0.079037
13 0.080871 0.085928 0.066116 0.070103 0.074432 0.079137 0.084251
14 0.086325 0.091979 0.069848 0.074248 0.079037 0.084251 0.089931
</pre></div>
</div>
</div>
@@ -1448,27 +1445,10 @@ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\bolds
<div class="section" id="code-for-svd-and-inversion-of-matrices">
<h2><span class="section-number">4.10. </span>Code for SVD and Inversion of Matrices<a class="headerlink" href="#code-for-svd-and-inversion-of-matrices" title="Permalink to this headline"></a></h2>
<p>How do we use the SVD to invert a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^\boldsymbol{X}\)</span> which is singular or near singular?
The simple answer is to use the linear algebra function for pseudoinvers, that is</p>
The simple answer is to use the linear algebra function for the pseudoinverse, that is</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">Ainv</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linlag</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">AttributeError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="o">&lt;</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">6</span><span class="o">-</span><span class="mi">52</span><span class="n">d2c51caad1</span><span class="o">&gt;</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span>
<span class="ne">----&gt; </span><span class="mi">1</span> <span class="n">Ainv</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linlag</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">A</span><span class="p">)</span>
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/numpy/__init__.py</span> in <span class="ni">__getattr__</span><span class="nt">(attr)</span>
<span class="g g-Whitespace"> </span><span class="mi">212</span> <span class="k">return</span> <span class="n">Tester</span>
<span class="g g-Whitespace"> </span><span class="mi">213</span> <span class="k">else</span><span class="p">:</span>
<span class="ne">--&gt; </span><span class="mi">214</span> <span class="k">raise</span> <span class="ne">AttributeError</span><span class="p">(</span><span class="s2">&quot;module </span><span class="si">{!r}</span><span class="s2"> has no attribute &quot;</span>
<span class="g g-Whitespace"> </span><span class="mi">215</span> <span class="s2">&quot;</span><span class="si">{!r}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">attr</span><span class="p">))</span>
<span class="g g-Whitespace"> </span><span class="mi">216</span>
<span class="ne">AttributeError</span>: module &#39;numpy&#39; has no attribute &#39;linlag&#39;
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1">#Ainv = np.linlag.pinv(A)</span>
</pre></div>
</div>
</div>
@@ -1508,6 +1488,24 @@ The simple answer is to use the linear algebra function for pseudoinvers, that i
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[1 2 3]
[2 4 5]
[3 5 6]]
test U
[[ 2.22044605e-16 -7.77156117e-16 -5.55111512e-16]
[-7.77156117e-16 0.00000000e+00 -1.11022302e-16]
[-5.55111512e-16 -1.11022302e-16 0.00000000e+00]]
test VT
[[ 1.11022302e-16 -2.22044605e-16 1.38777878e-16]
[-2.22044605e-16 -1.11022302e-16 -1.11022302e-16]
[ 1.38777878e-16 -1.11022302e-16 0.00000000e+00]]
[[2.35367281e-12 1.70885528e-12 3.20632410e-13]
[2.17248441e-12 1.46016532e-12 2.00728323e-13]
[6.95443703e-13 4.13891144e-13 2.13162821e-14]]
</pre></div>
</div>
</div>
</div>
<p>Although our matrix to invert <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> is a square matrix, our matrix may be singular.</p>
<p>The pseudoinverse is the generalization of the matrix inverse for square matrices to
@@ -1548,6 +1546,18 @@ It is used for the calculation of the inverse for singular or near singular matr
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[0.3 0.4]
[0.5 0.6]
[0.7 0.8]
[0.9 1. ]]
[[-13. -6. 1. 8. ]
[ 11.5 5.5 -0.5 -6.5]]
[[0. 0. 0. 0.]
[0. 0. 0. 0.]]
</pre></div>
</div>
</div>
</div>
<p>As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by <strong>Numpy</strong>.</p>
</div>
@@ -1894,6 +1904,14 @@ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\ve
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[2. 2.]
Training MSE for OLS
3.0
</pre></div>
</div>
<img alt="_images/chapter2_287_1.png" src="_images/chapter2_287_1.png" />
</div>
</div>
<p>We see here that we reach a plateau. What is actually happening?</p>
<div class="cell docutils container">
@@ -1957,6 +1975,214 @@ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\ve
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[2. 2.]
Training MSE for OLS
3.0
[1.99995 1.99980002]
[ 0.50001525 -0.99953475]
[1.99993978 1.99975913]
[ 0.50001525 -0.99944272]
[1.99992746 1.99970988]
[ 0.50001525 -0.99933188]
[1.99991263 1.99965056]
[ 0.50001525 -0.99919837]
[1.99989476 1.99957911]
[ 0.50001524 -0.99903755]
[1.99987324 1.99949306]
[ 0.50001524 -0.99884384]
[1.99984732 1.99938942]
[ 0.50001524 -0.99861053]
[1.9998161 1.99926459]
[ 0.50001523 -0.9983295 ]
[1.99977849 1.99911427]
[ 0.50001523 -0.99799099]
[1.9997332 1.99893323]
[ 0.50001522 -0.99758326]
[1.99967865 1.99871521]
[ 0.50001521 -0.99709215]
[1.99961294 1.99845267]
[ 0.50001521 -0.99650061]
[1.99953381 1.99813653]
[ 0.50001519 -0.99578809]
[1.9994385 1.99775587]
[ 0.50001518 -0.99492986]
[1.9993237 1.99729756]
[ 0.50001517 -0.99389612]
[1.99918546 1.9967458 ]
[ 0.50001515 -0.99265097]
[1.99901896 1.99608161]
[ 0.50001512 -0.99115119]
[1.99881845 1.99528218]
[ 0.5000151 -0.9893447]
[1.998577 1.9943201]
[ 0.50001506 -0.98716878]
[1.99828624 1.99316252]
[ 0.50001502 -0.98454786]
[1.99793613 1.99176998]
[ 0.50001498 -0.98139097]
[1.99751458 1.99009525]
[ 0.50001492 -0.97758848]
[1.99700706 1.98808176]
[ 0.50001485 -0.97300836]
[1.9963961 1.98566191]
[ 0.50001476 -0.9674916 ]
[1.99566069 1.98275501]
[ 0.50001466 -0.96084663]
[1.9947756 1.97926491]
[ 0.50001454 -0.95284275]
[1.99371056 1.97507735]
[ 0.50001439 -0.94320205]
[1.99242921 1.97005689]
[ 0.50001422 -0.93158979]
[1.99088801 1.9640435 ]
[ 0.500014 -0.91760278]
[1.9890348 1.95684892]
[ 0.50001374 -0.90075537]
[1.98680716 1.9482527 ]
[ 0.50001343 -0.88046261]
[1.98413059 1.93799826]
[ 0.50001306 -0.85601992]
[1.98091621 1.92578916]
[ 0.50001261 -0.8265786 ]
[1.97705827 1.91128596]
[ 0.50001207 -0.79111643]
[1.97243128 1.89410423]
[ 0.50001142 -0.74840212]
[1.96688672 1.87381451]
[ 0.50001063 -0.69695259]
[1.96024953 1.84994524]
[ 0.50000969 -0.63498144]
[1.95231424 1.82198978]
[ 0.50000855 -0.56033697]
[1.94284104 1.78941903]
[ 0.50000718 -0.47042744]
[1.93155188 1.75170092]
[ 0.50000553 -0.3621311 ]
[1.91812702 1.70832814]
[ 0.50001414 -0.23167717]
[1.90220243 1.65885453]
[ 0.50000455 -0.07456491]
[1.88336879 1.60293962]
[ 0.47132891 -0. ]
[1.86117291 1.54039921]
[ 0.41433969 -0. ]
[1.83512277 1.47125748]
[ 0.34569596 -0. ]
[1.80469739 1.39579407]
[ 0.26301436 -0. ]
[1.76936315 1.31457796]
[ 0.16342407 -0. ]
[1.72859758 1.22847924]
[ 0.04346721 -0. ]
[1.68192193 1.13865173]
[ 0. -0.]
[1.62894215 1.04648335]
[ 0. -0.]
[1.56939714 0.95351665]
[ 0. -0.]
[1.50321091 0.86134827]
[ 0. -0.]
[1.43054282 0.77152076]
[ 0. -0.]
[1.35182854 0.68542204]
[ 0. -0.]
[1.26780278 0.60420593]
[ 0. -0.]
[1.17949575 0.52874252]
[ 0. -0.]
[1.0881981 0.45960079]
[ 0. -0.]
[0.99539415 0.39706038]
[ 0. -0.]
[0.90266948 0.34114547]
[ 0. -0.]
[0.81160425 0.29167186]
[ 0. -0.]
[0.7236674 0.24829908]
[ 0. -0.]
[0.64012627 0.21058097]
[ 0. -0.]
[0.56198284 0.17801022]
[ 0. -0.]
[0.48994188 0.15005476]
[ 0. -0.]
[0.42441033 0.12618549]
[ 0. -0.]
[0.3655222 0.10589577]
[ 0. -0.]
[0.31318084 0.08871404]
[ 0. -0.]
[0.26710969 0.07421084]
[ 0. -0.]
[0.22690428 0.06200174]
[ 0. -0.]
[0.19207979 0.0517473 ]
[ 0. -0.]
[0.16211139 0.04315108]
[ 0. -0.]
[0.13646574 0.0359565 ]
[ 0. -0.]
[0.11462415 0.02994311]
[ 0. -0.]
[0.09609807 0.02492265]
[ 0. -0.]
[0.08043851 0.02073509]
[ 0. -0.]
[0.06724062 0.01724499]
[ 0. -0.]
[0.05614483 0.01433809]
[ 0. -0.]
[0.04683565 0.01191824]
[ 0. -0.]
[0.039039 0.00990475]
[ 0. -0.]
[0.03251863 0.00823002]
[ 0. -0.]
[0.02707227 0.00683748]
[ 0. -0.]
[0.02252765 0.0056799 ]
[ 0. -0.]
[0.01873869 0.00471782]
[ 0. -0.]
[0.01558197 0.00391839]
[ 0. -0.]
[0.01295356 0.0032542 ]
[ 0. -0.]
[0.01076611 0.00270244]
[ 0. -0.]
[0.00894639 0.00224413]
[ 0. -0.]
[0.0074331 0.00186347]
[ 0. -0.]
[0.00617499 0.00154733]
[ 0. -0.]
[0.00512927 0.00128479]
[ 0. -0.]
[0.00426027 0.00106677]
[ 0. -0.]
[0.00353823 0.00088573]
[ 0. -0.]
[0.00293838 0.00073541]
[ 0. -0.]
[0.0024401 0.00061058]
[ 0. -0.]
[0.00202624 0.00050694]
[ 0. -0.]
[0.00168251 0.00042089]
[ 0. -0.]
[0.00139705 0.00034944]
[ 0. -0.]
[0.00115999 0.00029012]
[ 0. -0.]
[0.00096314 0.00024087]
[ 0. -0.]
[0.00079968 0.00019998]
[ 0. -0.]
</pre></div>
</div>
<img alt="_images/chapter2_289_1.png" src="_images/chapter2_289_1.png" />
</div>
</div>
<p>Another Example, now with a polynomial fit.</p>
<div class="cell docutils container">
@@ -2042,6 +2268,16 @@ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\ve
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.03099776 -0.17917768 5.18029127]
Training MSE for OLS
0.009163470508352228
Test MSE OLS
0.008675369724976777
</pre></div>
</div>
<img alt="_images/chapter2_291_1.png" src="_images/chapter2_291_1.png" />
</div>
</div>
</div>
<div class="section" id="linking-the-regression-analysis-with-a-statistical-interpretation">
@@ -2420,6 +2656,22 @@ polynomial fit and that for larger and larger polynomial degrees of freedom, the
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.0169643 0.27924636 -1.4087793 1.03308408 0. ]
Test MSE OLS
0.958228616652075
0.001 [ 1.0170259 0.27852549 -1.40702 1.03193199 0. ]
0.001 [ 1.034342 -0.18063928 -0. 0. 0. ]
0.021544346900318832 [ 1.01825571 0.26412372 -1.37186301 1.00890601 0. ]
0.021544346900318832 [ 0.92280994 -0. -0. -0. 0. ]
0.46415888336127775 [ 1.0344707 0.07160764 -0.89928965 0.69843037 0. ]
0.46415888336127775 [0.48019541 0. 0. 0. 0. ]
10.0 [ 1.04529095 -0.18224665 -0.15751596 0.16831012 0. ]
10.0 [0. 0. 0. 0. 0.]
</pre></div>
</div>
<img alt="_images/chapter2_351_1.png" src="_images/chapter2_351_1.png" />
</div>
</div>
<p>How can we understand this?</p>
<p>Using Bayes’ theorem we can gain a better intuition about Ridge and Lasso regression.</p>
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/jupyter_execute/chapter2.ipynb
+419 -93
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/jupyter_execute/chapter2.py
+2 -2
View File
@@ -1162,9 +1162,9 @@ This equation does not lead to a nice analytical equation as in either Ridge reg
## Code for SVD and Inversion of Matrices
How do we use the SVD to invert a matrix $\boldsymbol{X}^\boldsymbol{X}$ which is singular or near singular?
The simple answer is to use the linear algebra function for pseudoinvers, that is
The simple answer is to use the linear algebra function for the pseudoinverse, that is
Ainv = np.linlag.pinv(A)
#Ainv = np.linlag.pinv(A)
Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD.
doc/LectureNotes/_build/jupyter_execute/chapter2_287_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.0 KiB

doc/LectureNotes/_build/jupyter_execute/chapter2_289_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

doc/LectureNotes/_build/jupyter_execute/chapter2_291_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

doc/LectureNotes/_build/jupyter_execute/chapter2_351_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 10 KiB

doc/LectureNotes/chapter2.ipynb
+2 -2
View File
@@ -2133,7 +2133,7 @@
"## Code for SVD and Inversion of Matrices\n",
"\n",
"How do we use the SVD to invert a matrix $\\boldsymbol{X}^\\boldsymbol{X}$ which is singular or near singular?\n",
"The simple answer is to use the linear algebra function for pseudoinvers, that is"
"The simple answer is to use the linear algebra function for the pseudoinverse, that is"
]
},
{
@@ -2145,7 +2145,7 @@
},
"outputs": [],
"source": [
"Ainv = np.linlag.pinv(A)"
"#Ainv = np.linlag.pinv(A)"
]
},
{