lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-10-08 17:04:26 +02:00
parent 87d1e4455c
commit dbbebedb06
114 changed files with 3736 additions and 1751 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/week40.html
+159 -125
View File
@@ -55,6 +55,7 @@ const thebe_selector_output = ".output, .cell_output"
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Exercises week 41" href="exercisesweek41.html" />
<link rel="prev" title="Week 39: Optimization and Gradient Methods" href="week39.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="None">
@@ -312,6 +313,33 @@ const thebe_selector_output = ".output, .cell_output"
Week 40: Gradient descent methods (continued) and start Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek41.html">
Exercises week 41
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week41.html">
Week 41 Neural networks and constructing a neural network code
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Projects
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="project1.html">
Project 1 on Machine Learning, deadline October 7 (midnight), 2024
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="project2.html">
Project 2 on Machine Learning, deadline November 4 (Midnight)
</a>
</li>
</ul>
</div>
@@ -1276,17 +1304,17 @@ We summarize some of these here for the methods we hvae studied in project one,
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Parameters for OLS using gradient descent
[[4.14721582]
[2.54760497]
[5.2212296 ]]
[[4.26604611]
[2.29234681]
[5.33329216]]
Parameters for Ridge using gradient descent
[[3.75424998]
[3.49608088]
[4.78010668]]
[[3.6161104 ]
[3.78762558]
[4.65410649]]
Parameters for Lasso using gradient descent
[[3.62716271]
[3.82146046]
[4.64516328]]
[[4.24335376]
[2.36219301]
[5.29669411]]
</pre></div>
</div>
</div>
@@ -1338,11 +1366,11 @@ Parameters for Lasso using gradient descent
[[4.]
[3.]
[5.]]
0 [-25.77106886] [-35.02189606]
1 [3.49285045e-13] [4.71505132e-13]
2 [1.24344979e-16] [3.86639832e-16]
3 [9.05941988e-16] [1.48786826e-15]
4 [-7.99360578e-16] [-1.35823372e-15]
0 [-26.28886314] [-34.64721597]
1 [-1.83231208e-13] [-1.80848093e-13]
2 [6.92779167e-16] [1.22835717e-15]
3 [-1.3500312e-15] [-1.8110093e-15]
4 [7.46069873e-16] [9.30442002e-16]
beta from own Newton code
[[4.]
[3.]
@@ -1603,9 +1631,6 @@ function.</p>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>gamma_j after 500 epochs: 9.97108e-05
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
</pre></div>
</div>
</div>
</div>
</div>
@@ -1687,15 +1712,15 @@ function.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
[[4.16180248]
[2.8250103 ]]
Eigenvalues of Hessian Matrix:[0.26646852 4.68110474]
[[4.36014743]
[2.76030841]]
Eigenvalues of Hessian Matrix:[0.36278226 3.73204357]
theta from own gd
[[4.16180248]
[2.8250103 ]]
[[4.36014743]
[2.76030841]]
theta from own sdg
[[4.16257872]
[2.80576215]]
[[4.34582863]
[2.81684805]]
</pre></div>
</div>
<img alt="_images/week40_34_1.png" src="_images/week40_34_1.png" />
@@ -2409,12 +2434,12 @@ first example shows results with ordinary leats squares.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
[[3.98246764]
[3.04018043]]
Eigenvalues of Hessian Matrix:[0.31541884 4.4735968 ]
[[3.85068028]
[3.09808149]]
Eigenvalues of Hessian Matrix:[0.31897935 3.95770298]
theta from own gd
[[3.98246764]
[3.04018043]]
[[3.85068028]
[3.09808149]]
</pre></div>
</div>
<img alt="_images/week40_100_1.png" src="_images/week40_100_1.png" />
@@ -2483,75 +2508,77 @@ theta from own gd
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
[[4.]
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.]
[3.]]
Eigenvalues of Hessian Matrix:[0.32799141 4.41229845]
0 [-16.27698328] [-18.5653087]
1 [-0.44867801] [0.37354121]
2 [-0.41532521] [0.34577375]
3 [-0.38445171] [0.32007041]
4 [-0.35587321] [0.29627774]
5 [-0.32941912] [0.27425372]
6 [-0.30493151] [0.25386687]
7 [-0.2822642] [0.23499549]
8 [-0.26128189] [0.21752693]
9 [-0.24185931] [0.20135691]
10 [-0.22388052] [0.1863889]
11 [-0.2072382] [0.17253354]
12 [-0.191833] [0.15970814]
13 [-0.17757295] [0.14783612]
14 [-0.16437294] [0.13684661]
15 [-0.15215415] [0.12667402]
16 [-0.14084366] [0.11725761]
17 [-0.13037395] [0.10854118]
18 [-0.12068251] [0.1004727]
19 [-0.11171148] [0.09300398]
20 [-0.10340733] [0.08609047]
21 [-0.09572047] [0.07969087]
22 [-0.08860502] [0.07376699]
23 [-0.0820185] [0.06828347]
24 [-0.0759216] [0.06320757]
25 [-0.07027791] [0.05850899]
26 [-0.06505375] [0.05415968]
27 [-0.06021793] [0.05013368]
28 [-0.05574159] [0.04640695]
29 [-0.051598] [0.04295726]
Eigenvalues of Hessian Matrix:[0.35713539 3.88632765]
0 [-9.01615836] [-9.6932681]
1 [0.01454469] [-0.01357366]
2 [0.0132081] [-0.0123263]
3 [0.01199434] [-0.01119357]
4 [0.01089212] [-0.01016493]
5 [0.00989118] [-0.00923082]
6 [0.00898223] [-0.00838255]
7 [0.0081568] [-0.00761224]
8 [0.00740723] [-0.00691271]
9 [0.00672654] [-0.00627746]
10 [0.0061084] [-0.00570059]
11 [0.00554707] [-0.00517673]
12 [0.00503732] [-0.00470102]
13 [0.00457441] [-0.00426902]
14 [0.00415405] [-0.00387671]
15 [0.00377231] [-0.00352046]
16 [0.00342565] [-0.00319695]
17 [0.00311085] [-0.00290316]
18 [0.00282498] [-0.00263638]
19 [0.00256537] [-0.0023941]
20 [0.00232963] [-0.0021741]
21 [0.00211555] [-0.00197431]
22 [0.00192114] [-0.00179288]
23 [0.00174459] [-0.00162812]
24 [0.00158427] [-0.0014785]
25 [0.00143869] [-0.00134264]
26 [0.00130648] [-0.00121925]
27 [0.00118642] [-0.00110721]
28 [0.00107739] [-0.00100546]
29 [0.00097839] [-0.00091307]
theta from own gd
[[3.85437905]
[3.12123488]]
0 [-0.04776242] [0.039764]
1 [-0.04421197] [0.03680811]
2 [-0.0398603] [0.03318519]
3 [-0.03559176] [0.02963147]
4 [-0.03166546] [0.02636268]
5 [-0.02813369] [0.02342235]
6 [-0.02498282] [0.02079913]
7 [-0.02218045] [0.01846605]
8 [-0.01969093] [0.01639344]
9 [-0.01748034] [0.01455304]
10 [-0.01551775] [0.0129191]
11 [-0.01377545] [0.01146857]
12 [-0.01222875] [0.01018089]
13 [-0.01085571] [0.00903778]
14 [-0.00963683] [0.00802302]
15 [-0.0085548] [0.00712219]
16 [-0.00759427] [0.00632251]
17 [-0.00674158] [0.00561262]
18 [-0.00598464] [0.00498243]
19 [-0.00531268] [0.004423]
20 [-0.00471617] [0.00392639]
21 [-0.00418664] [0.00348553]
22 [-0.00371656] [0.00309418]
23 [-0.00329927] [0.00274676]
24 [-0.00292882] [0.00243835]
25 [-0.00259997] [0.00216458]
26 [-0.00230805] [0.00192154]
27 [-0.0020489] [0.00170579]
28 [-0.00181885] [0.00151426]
29 [-0.00161463] [0.00134424]
[[4.00248779]
[2.9976783 ]]
0 [0.00088848] [-0.00082916]
1 [0.00080683] [-0.00075296]
2 [0.00070819] [-0.00066091]
3 [0.00061352] [-0.00057256]
4 [0.00052874] [-0.00049344]
5 [0.00045472] [-0.00042436]
6 [0.00039072] [-0.00036464]
7 [0.00033562] [-0.00031321]
8 [0.00028825] [-0.000269]
9 [0.00024755] [-0.00023102]
10 [0.00021259] [-0.0001984]
11 [0.00018256] [-0.00017038]
12 [0.00015678] [-0.00014631]
13 [0.00013464] [-0.00012565]
14 [0.00011562] [-0.0001079]
15 [9.92929225e-05] [-9.26639089e-05]
16 [8.52694246e-05] [-7.95766504e-05]
17 [7.32265127e-05] [-6.83377498e-05]
18 [6.28844641e-05] [-5.86861591e-05]
19 [5.40030606e-05] [-5.03976976e-05]
20 [4.63760101e-05] [-4.32798457e-05]
21 [3.98261559e-05] [-3.71672742e-05]
22 [3.42013616e-05] [-3.19180036e-05]
23 [2.93709777e-05] [-2.74101067e-05]
24 [2.52228066e-05] [-2.35388766e-05]
25 [2.1660497e-05] [-2.02143946e-05]
26 [1.86013055e-05] [-1.73594414e-05]
27 [1.59741748e-05] [-1.49077038e-05]
28 [1.37180834e-05] [-1.2802234e-05]
29 [1.17806281e-05] [-1.09941275e-05]
theta from own gd wth momentum
[[3.99562995]
[3.00363823]]
[[4.00002833]
[2.99997356]]
</pre></div>
</div>
</div>
@@ -2640,18 +2667,20 @@ theta from own gd wth momentum
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
[[4.10426556]
[2.93942872]]
Eigenvalues of Hessian Matrix:[0.31367041 4.07517385]
theta from own gd
[[4.10426556]
[2.93942872]]
[[3.91650453]
[2.94495682]]
Eigenvalues of Hessian Matrix:[0.34862407 4.10453899]
</pre></div>
</div>
<img alt="_images/week40_104_1.png" src="_images/week40_104_1.png" />
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own gd
[[3.91650453]
[2.94495682]]
</pre></div>
</div>
<img alt="_images/week40_104_2.png" src="_images/week40_104_2.png" />
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own sdg
[[4.08587209]
[2.96301764]]
[[3.8901199 ]
[2.92458892]]
</pre></div>
</div>
</div>
@@ -2733,17 +2762,15 @@ theta from own gd
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
[[4.33528448]
[2.81889188]]
Eigenvalues of Hessian Matrix:[0.30916402 4.51611732]
[[4.05785974]
[2.95842106]]
Eigenvalues of Hessian Matrix:[0.29678339 4.37215356]
theta from own gd
[[4.33477019]
[2.81931348]]
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own sdg with momentum
[[4.40800396]
[2.78459458]]
[[4.05738629]
[2.95882223]]
theta from own sdg with momentum
[[4.07489511]
[2.90281987]]
</pre></div>
</div>
</div>
@@ -2812,9 +2839,9 @@ theta from own gd
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own AdaGrad
[[2.00003828]
[2.99979896]
[4.00019258]]
[[1.99994537]
[3.00034209]
[3.99966798]]
</pre></div>
</div>
</div>
@@ -2890,9 +2917,9 @@ theta from own gd
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own RMSprop
[[2.01129731]
[3.01249445]
[4.00885858]]
[[1.99975636]
[3.00348281]
[3.99607299]]
</pre></div>
</div>
</div>
@@ -2972,9 +2999,9 @@ theta from own gd
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own ADAM
[[2.00001716]
[2.99992107]
[4.00007706]]
[[2.00002276]
[2.99985884]
[4.00009004]]
</pre></div>
</div>
</div>
@@ -3095,7 +3122,7 @@ It provides composable transformations of Python+NumPy programs: differentiate,
return asarray(x, dtype=self.dtype)
</pre></div>
</div>
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x11753f700&gt;]
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x11bcee130&gt;]
</pre></div>
</div>
<img alt="_images/week40_120_2.png" src="_images/week40_120_2.png" />
@@ -3130,7 +3157,7 @@ It provides composable transformations of Python+NumPy programs: differentiate,
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.collections.PathCollection at 0x11bc3a1f0&gt;
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.collections.PathCollection at 0x11bdd4640&gt;
</pre></div>
</div>
<img alt="_images/week40_122_1.png" src="_images/week40_122_1.png" />
@@ -3810,6 +3837,13 @@ become the most popular for <em>deep neural networks</em></p>
<p class="prev-next-title">Week 39: Optimization and Gradient Methods</p>
</div>
</a>
<a class='right-next' id="next-link" href="exercisesweek41.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Exercises week 41</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
</div>
</div>