lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-10-21 13:13:08 +02:00
parent 4afc70996e
commit 3efc0623f6
15 changed files with 4471 additions and 3735 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/week43.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/_sources/week43.ipynb
+1329 -662
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/html/week43.html
+90 -520
View File
@@ -711,13 +711,13 @@ const thebe_selector_output = ".output, .cell_output"
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#solving-odes-with-deep-learning">
Solving ODEs with Deep Learning
<a class="reference internal nav-link" href="#solving-differential-equations-with-deep-learning">
Solving differential equations with Deep Learning
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#ordinary-differential-equations">
Ordinary Differential Equations
<a class="reference internal nav-link" href="#ordinary-differential-equations-first">
Ordinary Differential Equations first
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -1250,13 +1250,13 @@ const thebe_selector_output = ".output, .cell_output"
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#solving-odes-with-deep-learning">
Solving ODEs with Deep Learning
<a class="reference internal nav-link" href="#solving-differential-equations-with-deep-learning">
Solving differential equations with Deep Learning
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#ordinary-differential-equations">
Ordinary Differential Equations
<a class="reference internal nav-link" href="#ordinary-differential-equations-first">
Ordinary Differential Equations first
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -2346,7 +2346,7 @@ the <em>Hadamard product</em>, meaning element-wise multiplication.</p>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Old accuracy on training data: 0.1440501043841336
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2682,7 +2682,7 @@ Lambda = 10.0
Accuracy score on test set: 0.19166666666666668
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2691,7 +2691,7 @@ Lambda = 1e-05
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2700,7 +2700,7 @@ Lambda = 0.0001
Accuracy score on test set: 0.08611111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2709,7 +2709,7 @@ Lambda = 0.001
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2718,7 +2718,7 @@ Lambda = 0.01
Accuracy score on test set: 0.08888888888888889
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2727,7 +2727,7 @@ Lambda = 0.1
Accuracy score on test set: 0.08611111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2736,7 +2736,7 @@ Lambda = 1.0
Accuracy score on test set: 0.08888888888888889
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2745,11 +2745,11 @@ Lambda = 10.0
Accuracy score on test set: 0.09166666666666666
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2758,11 +2758,11 @@ Lambda = 1e-05
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2771,11 +2771,11 @@ Lambda = 0.0001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2784,11 +2784,11 @@ Lambda = 0.001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2797,130 +2797,37 @@ Lambda = 0.01
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 0.1
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 1.0
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 10.0
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 1e-05
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 0.0001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 0.001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 0.01
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 0.1
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 1.0
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 10.0
Accuracy score on test set: 0.07777777777777778
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">KeyboardInterrupt</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="n">Cell</span> <span class="n">In</span><span class="p">[</span><span class="mi">8</span><span class="p">],</span> <span class="n">line</span> <span class="mi">11</span>
<span class="g g-Whitespace"> </span><span class="mi">8</span> <span class="k">for</span> <span class="n">j</span><span class="p">,</span> <span class="n">lmbd</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">lmbd_vals</span><span class="p">):</span>
<span class="g g-Whitespace"> </span><span class="mi">9</span> <span class="n">dnn</span> <span class="o">=</span> <span class="n">NeuralNetwork</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">Y_train_onehot</span><span class="p">,</span> <span class="n">eta</span><span class="o">=</span><span class="n">eta</span><span class="p">,</span> <span class="n">lmbd</span><span class="o">=</span><span class="n">lmbd</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="n">epochs</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">10</span> <span class="n">n_hidden_neurons</span><span class="o">=</span><span class="n">n_hidden_neurons</span><span class="p">,</span> <span class="n">n_categories</span><span class="o">=</span><span class="n">n_categories</span><span class="p">)</span>
<span class="ne">---&gt; </span><span class="mi">11</span> <span class="n">dnn</span><span class="o">.</span><span class="n">train</span><span class="p">()</span>
<span class="g g-Whitespace"> </span><span class="mi">13</span> <span class="n">DNN_numpy</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">dnn</span>
<span class="g g-Whitespace"> </span><span class="mi">15</span> <span class="n">test_predict</span> <span class="o">=</span> <span class="n">dnn</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="nn">Cell In[6], line 99,</span> in <span class="ni">NeuralNetwork.train</span><span class="nt">(self)</span>
<span class="g g-Whitespace"> </span><span class="mi">96</span> <span class="bp">self</span><span class="o">.</span><span class="n">Y_data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">Y_data_full</span><span class="p">[</span><span class="n">chosen_datapoints</span><span class="p">]</span>
<span class="g g-Whitespace"> </span><span class="mi">98</span> <span class="bp">self</span><span class="o">.</span><span class="n">feed_forward</span><span class="p">()</span>
<span class="ne">---&gt; </span><span class="mi">99</span> <span class="bp">self</span><span class="o">.</span><span class="n">backpropagation</span><span class="p">()</span>
<span class="nn">Cell In[6], line 64,</span> in <span class="ni">NeuralNetwork.backpropagation</span><span class="nt">(self)</span>
<span class="g g-Whitespace"> </span><span class="mi">61</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_weights_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">a_h</span><span class="o">.</span><span class="n">T</span><span class="p">,</span> <span class="n">error_output</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">62</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_bias_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">error_output</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="ne">---&gt; </span><span class="mi">64</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_weights_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">X_data</span><span class="o">.</span><span class="n">T</span><span class="p">,</span> <span class="n">error_hidden</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">65</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_bias_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">error_hidden</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">67</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">lmbd</span> <span class="o">&gt;</span> <span class="mf">0.0</span><span class="p">:</span>
<span class="ne">KeyboardInterrupt</span>:
</pre></div>
</div>
</div>
@@ -2966,22 +2873,6 @@ Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_87605/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<img alt="_images/week43_96_1.png" src="_images/week43_96_1.png" />
<img alt="_images/week43_96_2.png" src="_images/week43_96_2.png" />
</div>
</div>
</div>
<div class="section" id="scikit-learn-implementation">
@@ -3017,328 +2908,6 @@ performance overall.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
Lambda = 1e-05
Accuracy score on test set: 0.18333333333333332
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
Lambda = 0.0001
Accuracy score on test set: 0.18611111111111112
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
Lambda = 0.001
Accuracy score on test set: 0.13055555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
Lambda = 0.01
Accuracy score on test set: 0.24444444444444444
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
Lambda = 0.1
Accuracy score on test set: 0.23333333333333334
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
Lambda = 1.0
Accuracy score on test set: 0.12777777777777777
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
Lambda = 10.0
Accuracy score on test set: 0.1527777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
Lambda = 1e-05
Accuracy score on test set: 0.9111111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
Lambda = 0.0001
Accuracy score on test set: 0.8888888888888888
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
Lambda = 0.001
Accuracy score on test set: 0.8722222222222222
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
Lambda = 0.01
Accuracy score on test set: 0.8305555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
Lambda = 0.1
Accuracy score on test set: 0.8888888888888888
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
Lambda = 1.0
Accuracy score on test set: 0.8805555555555555
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
Lambda = 10.0
Accuracy score on test set: 0.8944444444444445
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
Lambda = 1e-05
Accuracy score on test set: 0.975
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
Lambda = 0.0001
Accuracy score on test set: 0.9777777777777777
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
Lambda = 0.001
Accuracy score on test set: 0.9805555555555555
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
Lambda = 0.01
Accuracy score on test set: 0.9861111111111112
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
Lambda = 0.1
Accuracy score on test set: 0.9805555555555555
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
Lambda = 1.0
Accuracy score on test set: 0.9777777777777777
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
Lambda = 10.0
Accuracy score on test set: 0.9444444444444444
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
Lambda = 1e-05
Accuracy score on test set: 0.9861111111111112
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
Lambda = 0.0001
Accuracy score on test set: 0.9888888888888889
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
Lambda = 0.001
Accuracy score on test set: 0.9888888888888889
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
Lambda = 0.01
Accuracy score on test set: 0.9861111111111112
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
Lambda = 0.1
Accuracy score on test set: 0.9888888888888889
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
Lambda = 1.0
Accuracy score on test set: 0.9722222222222222
Learning rate = 0.01
Lambda = 10.0
Accuracy score on test set: 0.9527777777777777
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 1e-05
Accuracy score on test set: 0.9027777777777778
Learning rate = 0.1
Lambda = 0.0001
Accuracy score on test set: 0.8583333333333333
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 0.001
Accuracy score on test set: 0.8722222222222222
Learning rate = 0.1
Lambda = 0.01
Accuracy score on test set: 0.9055555555555556
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 0.1
Accuracy score on test set: 0.8805555555555555
Learning rate = 0.1
Lambda = 1.0
Accuracy score on test set: 0.8722222222222222
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 10.0
Accuracy score on test set: 0.8666666666666667
Learning rate = 1.0
Lambda = 1e-05
Accuracy score on test set: 0.08611111111111111
Learning rate = 1.0
Lambda = 0.0001
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 0.001
Accuracy score on test set: 0.10555555555555556
Learning rate = 1.0
Lambda = 0.01
Accuracy score on test set: 0.17777777777777778
Learning rate = 1.0
Lambda = 0.1
Accuracy score on test set: 0.08333333333333333
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 1.0
Accuracy score on test set: 0.08888888888888889
Learning rate = 1.0
Lambda = 10.0
Accuracy score on test set: 0.09444444444444444
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 1e-05
Accuracy score on test set: 0.17222222222222222
Learning rate = 10.0
Lambda = 0.0001
Accuracy score on test set: 0.11666666666666667
Learning rate = 10.0
Lambda = 0.001
Accuracy score on test set: 0.10555555555555556
Learning rate = 10.0
Lambda = 0.01
Accuracy score on test set: 0.1388888888888889
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 0.1
Accuracy score on test set: 0.11388888888888889
Learning rate = 10.0
Lambda = 1.0
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 10.0
Accuracy score on test set: 0.09444444444444444
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="id1">
@@ -3382,10 +2951,6 @@ Accuracy score on test set: 0.09444444444444444
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/week43_100_0.png" src="_images/week43_100_0.png" />
<img alt="_images/week43_100_1.png" src="_images/week43_100_1.png" />
</div>
</div>
</div>
<div class="section" id="building-neural-networks-in-tensorflow-and-keras">
@@ -3424,14 +2989,6 @@ how simple solving a machine learning problem can be.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span> <span class="n">Cell</span> <span class="n">In</span><span class="p">[</span><span class="mi">12</span><span class="p">],</span> <span class="n">line</span> <span class="mi">1</span>
<span class="n">pip3</span> <span class="n">install</span> <span class="n">tensorflow</span>
<span class="o">^</span>
<span class="ne">SyntaxError</span>: invalid syntax
</pre></div>
</div>
</div>
</div>
<p>and/or if you use <strong>anaconda</strong>, just write (or install from the graphical user interface)
(current release of CPU-only TensorFlow)</p>
@@ -3558,7 +3115,7 @@ If you have Anaconda installed you may run the following command</p>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">n_neurons_layer2</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;sigmoid&#39;</span><span class="p">,</span> <span class="n">kernel_regularizer</span><span class="o">=</span><span class="n">regularizers</span><span class="o">.</span><span class="n">l2</span><span class="p">(</span><span class="n">lmbd</span><span class="p">)))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">n_categories</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;softmax&#39;</span><span class="p">))</span>
<span class="n">sgd</span> <span class="o">=</span> <span class="n">optimizers</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">lr</span><span class="o">=</span><span class="n">eta</span><span class="p">)</span>
<span class="n">sgd</span> <span class="o">=</span> <span class="n">optimizers</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="n">eta</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s1">&#39;categorical_crossentropy&#39;</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="n">sgd</span><span class="p">,</span> <span class="n">metrics</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;accuracy&#39;</span><span class="p">])</span>
<span class="k">return</span> <span class="n">model</span>
@@ -3738,7 +3295,7 @@ If you have Anaconda installed you may run the following command</p>
<span class="k">else</span><span class="p">:</span> <span class="c1">#Subsequent layers are capable of automatic shape inferencing</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="n">n_neuron</span><span class="p">,</span><span class="n">activation</span><span class="o">=</span><span class="s1">&#39;relu&#39;</span><span class="p">,</span><span class="n">kernel_regularizer</span><span class="o">=</span><span class="n">regularizers</span><span class="o">.</span><span class="n">l2</span><span class="p">(</span><span class="n">lamda</span><span class="p">)))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="n">activation</span><span class="o">=</span><span class="s1">&#39;softmax&#39;</span><span class="p">))</span> <span class="c1">#2 outputs - ordered and disordered (softmax for prob)</span>
<span class="n">sgd</span><span class="o">=</span><span class="n">optimizers</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">lr</span><span class="o">=</span><span class="n">eta</span><span class="p">)</span>
<span class="n">sgd</span><span class="o">=</span><span class="n">optimizers</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="n">eta</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s1">&#39;categorical_crossentropy&#39;</span><span class="p">,</span><span class="n">optimizer</span><span class="o">=</span><span class="n">sgd</span><span class="p">,</span><span class="n">metrics</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;accuracy&#39;</span><span class="p">])</span>
<span class="k">return</span> <span class="n">model</span>
@@ -4828,8 +4385,8 @@ digits between the range of 0 to 9.</p>
</div>
<p>Not bad, but the results depend strongly on the learning reate. Try different learning rates.</p>
</div>
<div class="section" id="solving-odes-with-deep-learning">
<h2>Solving ODEs with Deep Learning<a class="headerlink" href="#solving-odes-with-deep-learning" title="Permalink to this headline"></a></h2>
<div class="section" id="solving-differential-equations-with-deep-learning">
<h2>Solving differential equations with Deep Learning<a class="headerlink" href="#solving-differential-equations-with-deep-learning" title="Permalink to this headline"></a></h2>
<p>The Universal Approximation Theorem states that a neural network can
approximate any function at a single hidden layer along with one input
and output layer to any given precision.</p>
@@ -4841,8 +4398,8 @@ and output layer to any given precision.</p>
<p>The lectures on differential equations were developed by Kristine Baluka Hein, now PhD student at IFI.
A great thanks to Kristine.</p>
</div>
<div class="section" id="ordinary-differential-equations">
<h2>Ordinary Differential Equations<a class="headerlink" href="#ordinary-differential-equations" title="Permalink to this headline"></a></h2>
<div class="section" id="ordinary-differential-equations-first">
<h2>Ordinary Differential Equations first<a class="headerlink" href="#ordinary-differential-equations-first" title="Permalink to this headline"></a></h2>
<p>An ordinary differential equation (ODE) is an equation involving functions having one variable.</p>
<p>In general, an ordinary differential equation looks like</p>
<!-- Equation labels as ordinary links -->
@@ -5350,8 +4907,8 @@ P_{\text{output},\text{new}} &amp;= P_{\text{output}} - \lambda \nabla_{P_{\text
<span class="c1"># but with number of hidden layers specified by the user.</span>
<span class="k">def</span> <span class="nf">deep_neural_network</span><span class="p">(</span><span class="n">deep_params</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="c1"># N_hidden is the number of hidden layers</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consists of</span>
<span class="c1"># deep_params is a list, len() should be used</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consists of</span>
<span class="c1"># parameters to all the hidden</span>
<span class="c1"># layers AND the output layer.</span>
@@ -5572,9 +5129,12 @@ g(t) = \frac{Ag_0}{g_0 + (A - g_0)\exp(-\alpha A t)}
<span class="n">g0</span> <span class="o">=</span> <span class="mf">1.2</span>
<span class="k">return</span> <span class="n">alpha</span><span class="p">,</span> <span class="n">A</span><span class="p">,</span> <span class="n">g0</span>
<span class="k">def</span> <span class="nf">deep_neural_network</span><span class="p">(</span><span class="n">P</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">deep_neural_network</span><span class="p">(</span><span class="n">deep_params</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="c1"># N_hidden is the number of hidden layers</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">P</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="c1"># deep_params is a list, len() should be used</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consists of</span>
<span class="c1"># parameters to all the hidden</span>
<span class="c1"># layers AND the output layer.</span>
<span class="c1"># Assumes input x being an one-dimensional array</span>
<span class="n">num_values</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
@@ -5591,7 +5151,7 @@ g(t) = \frac{Ag_0}{g_0 + (A - g_0)\exp(-\alpha A t)}
<span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N_hidden</span><span class="p">):</span>
<span class="c1"># From the list of parameters P; find the correct weigths and bias for this layer</span>
<span class="n">w_hidden</span> <span class="o">=</span> <span class="n">P</span><span class="p">[</span><span class="n">l</span><span class="p">]</span>
<span class="n">w_hidden</span> <span class="o">=</span> <span class="n">deep_params</span><span class="p">[</span><span class="n">l</span><span class="p">]</span>
<span class="c1"># Add a row of ones to include bias</span>
<span class="n">x_prev</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">((</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span><span class="n">num_values</span><span class="p">)),</span> <span class="n">x_prev</span> <span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span>
@@ -5605,7 +5165,7 @@ g(t) = \frac{Ag_0}{g_0 + (A - g_0)\exp(-\alpha A t)}
<span class="c1">## Output layer:</span>
<span class="c1"># Get the weights and bias for this layer</span>
<span class="n">w_output</span> <span class="o">=</span> <span class="n">P</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="n">w_output</span> <span class="o">=</span> <span class="n">deep_params</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="c1"># Include bias:</span>
<span class="n">x_prev</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">((</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span><span class="n">num_values</span><span class="p">)),</span> <span class="n">x_prev</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span>
@@ -5616,6 +5176,8 @@ g(t) = \frac{Ag_0}{g_0 + (A - g_0)\exp(-\alpha A t)}
<span class="k">return</span> <span class="n">x_output</span>
<span class="k">def</span> <span class="nf">cost_function_deep</span><span class="p">(</span><span class="n">P</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="c1"># Evaluate the trial function with the current parameters P</span>
@@ -5906,7 +5468,10 @@ g(x) = x(1 - x)\exp(x)
<span class="k">def</span> <span class="nf">deep_neural_network</span><span class="p">(</span><span class="n">deep_params</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="c1"># N_hidden is the number of hidden layers</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="c1"># deep_params is a list, len() should be used</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consists of</span>
<span class="c1"># parameters to all the hidden</span>
<span class="c1"># layers AND the output layer.</span>
<span class="c1"># Assumes input x being an one-dimensional array</span>
<span class="n">num_values</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
@@ -5947,6 +5512,7 @@ g(x) = x(1 - x)\exp(x)
<span class="k">return</span> <span class="n">x_output</span>
<span class="k">def</span> <span class="nf">solve_ode_deep_neural_network</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">num_neurons</span><span class="p">,</span> <span class="n">num_iter</span><span class="p">,</span> <span class="n">lmb</span><span class="p">):</span>
<span class="c1"># num_hidden_neurons is now a list of number of neurons within each hidden layer</span>
@@ -6146,7 +5712,10 @@ f(x_{N_x - 2})
<span class="k">def</span> <span class="nf">deep_neural_network</span><span class="p">(</span><span class="n">deep_params</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="c1"># N_hidden is the number of hidden layers</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="c1"># deep_params is a list, len() should be used</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consists of</span>
<span class="c1"># parameters to all the hidden</span>
<span class="c1"># layers AND the output layer.</span>
<span class="c1"># Assumes input x being an one-dimensional array</span>
<span class="n">num_values</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
@@ -6187,6 +5756,7 @@ f(x_{N_x - 2})
<span class="k">return</span> <span class="n">x_output</span>
<span class="k">def</span> <span class="nf">solve_ode_deep_neural_network</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">num_neurons</span><span class="p">,</span> <span class="n">num_iter</span><span class="p">,</span> <span class="n">lmb</span><span class="p">):</span>
<span class="c1"># num_hidden_neurons is now a list of number of neurons within each hidden layer</span>
@@ -6456,7 +6026,7 @@ network at each possible pair <span class="math notranslate nohighlight">\((x,t)
<span class="n">num_points</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># N_hidden is the number of hidden layers</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="c1"># Assume that the input layer does nothing to the input x</span>
<span class="n">x_input</span> <span class="o">=</span> <span class="n">x</span>
@@ -6607,7 +6177,7 @@ Using TensorFlow results in a much better execution time. Try it!</p>
<span class="n">num_points</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># N_hidden is the number of hidden layers</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="c1"># Assume that the input layer does nothing to the input x</span>
<span class="n">x_input</span> <span class="o">=</span> <span class="n">x</span>
@@ -6750,7 +6320,7 @@ Using TensorFlow results in a much better execution time. Try it!</p>
<span class="n">T</span><span class="p">,</span><span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">meshgrid</span><span class="p">(</span><span class="n">t</span><span class="p">,</span><span class="n">x</span><span class="p">)</span>
<span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">))</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">gca</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_suplot</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s1">&#39;Solution from the deep neural network w/ </span><span class="si">%d</span><span class="s1"> layer&#39;</span><span class="o">%</span><span class="k">len</span>(num_hidden_neurons))
<span class="n">s</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">plot_surface</span><span class="p">(</span><span class="n">T</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">g_dnn_ag</span><span class="p">,</span><span class="n">linewidth</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">antialiased</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span><span class="n">cmap</span><span class="o">=</span><span class="n">cm</span><span class="o">.</span><span class="n">viridis</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s1">&#39;Time $t$&#39;</span><span class="p">)</span>
@@ -6758,14 +6328,14 @@ Using TensorFlow results in a much better execution time. Try it!</p>
<span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">))</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">gca</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_suplot</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s1">&#39;Analytical solution&#39;</span><span class="p">)</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">plot_surface</span><span class="p">(</span><span class="n">T</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">G_analytical</span><span class="p">,</span><span class="n">linewidth</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">antialiased</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span><span class="n">cmap</span><span class="o">=</span><span class="n">cm</span><span class="o">.</span><span class="n">viridis</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s1">&#39;Time $t$&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_ylabel</span><span class="p">(</span><span class="s1">&#39;Position $x$&#39;</span><span class="p">);</span>
<span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">))</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">gca</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_suplot</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s1">&#39;Difference&#39;</span><span class="p">)</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">plot_surface</span><span class="p">(</span><span class="n">T</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">diff_ag</span><span class="p">,</span><span class="n">linewidth</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">antialiased</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span><span class="n">cmap</span><span class="o">=</span><span class="n">cm</span><span class="o">.</span><span class="n">viridis</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s1">&#39;Time $t$&#39;</span><span class="p">)</span>
@@ -6943,7 +6513,7 @@ g(x,t) = \sin(\pi x)\cos(\pi t) - \sin(\pi x)\sin(\pi t)
<span class="n">num_points</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># N_hidden is the number of hidden layers</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="n">N_hidden</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">deep_params</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> <span class="c1"># -1 since params consist of parameters to all the hidden layers AND the output layer</span>
<span class="c1"># Assume that the input layer does nothing to the input x</span>
<span class="n">x_input</span> <span class="o">=</span> <span class="n">x</span>
@@ -7048,7 +6618,7 @@ g(x,t) = \sin(\pi x)\cos(\pi t) - \sin(\pi x)\sin(\pi t)
<span class="n">T</span><span class="p">,</span><span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">meshgrid</span><span class="p">(</span><span class="n">t</span><span class="p">,</span><span class="n">x</span><span class="p">)</span>
<span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">))</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">gca</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_suplot</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s1">&#39;Solution from the deep neural network w/ </span><span class="si">%d</span><span class="s1"> layer&#39;</span><span class="o">%</span><span class="k">len</span>(num_hidden_neurons))
<span class="n">s</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">plot_surface</span><span class="p">(</span><span class="n">T</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">res</span><span class="p">,</span><span class="n">linewidth</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">antialiased</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span><span class="n">cmap</span><span class="o">=</span><span class="n">cm</span><span class="o">.</span><span class="n">viridis</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s1">&#39;Time $t$&#39;</span><span class="p">)</span>
@@ -7056,7 +6626,7 @@ g(x,t) = \sin(\pi x)\cos(\pi t) - \sin(\pi x)\sin(\pi t)
<span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">))</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">gca</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_suplot</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s1">&#39;Analytical solution&#39;</span><span class="p">)</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">plot_surface</span><span class="p">(</span><span class="n">T</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">res_analytical</span><span class="p">,</span><span class="n">linewidth</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">antialiased</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span><span class="n">cmap</span><span class="o">=</span><span class="n">cm</span><span class="o">.</span><span class="n">viridis</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s1">&#39;Time $t$&#39;</span><span class="p">)</span>
@@ -7064,7 +6634,7 @@ g(x,t) = \sin(\pi x)\cos(\pi t) - \sin(\pi x)\sin(\pi t)
<span class="n">fig</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">))</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">gca</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_suplot</span><span class="p">(</span><span class="n">projection</span><span class="o">=</span><span class="s1">&#39;3d&#39;</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="s1">&#39;Difference&#39;</span><span class="p">)</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">plot_surface</span><span class="p">(</span><span class="n">T</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">diff</span><span class="p">,</span><span class="n">linewidth</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">antialiased</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span><span class="n">cmap</span><span class="o">=</span><span class="n">cm</span><span class="o">.</span><span class="n">viridis</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s1">&#39;Time $t$&#39;</span><span class="p">)</span>
doc/LectureNotes/_build/jupyter_execute/week43.ipynb
+1365 -1548
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/jupyter_execute/week43.py
+34 -21
View File
@@ -1366,7 +1366,7 @@ def create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories
model.add(Dense(n_neurons_layer2, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))
model.add(Dense(n_categories, activation='softmax'))
sgd = optimizers.SGD(lr=eta)
sgd = optimizers.SGD(learning_rate=eta)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
return model
@@ -1542,7 +1542,7 @@ def NN_model(inputsize,n_layers,n_neuron,eta,lamda):
else: #Subsequent layers are capable of automatic shape inferencing
model.add(Dense(n_neuron,activation='relu',kernel_regularizer=regularizers.l2(lamda)))
model.add(Dense(2,activation='softmax')) #2 outputs - ordered and disordered (softmax for prob)
sgd=optimizers.SGD(lr=eta)
sgd=optimizers.SGD(learning_rate=eta)
model.compile(loss='categorical_crossentropy',optimizer=sgd,metrics=['accuracy'])
return model
@@ -2623,7 +2623,7 @@ scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)
# Not bad, but the results depend strongly on the learning reate. Try different learning rates.
# ## Solving ODEs with Deep Learning
# ## Solving differential equations with Deep Learning
#
# The Universal Approximation Theorem states that a neural network can
# approximate any function at a single hidden layer along with one input
@@ -2642,7 +2642,7 @@ scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)
# The lectures on differential equations were developed by Kristine Baluka Hein, now PhD student at IFI.
# A great thanks to Kristine.
# ## Ordinary Differential Equations
# ## Ordinary Differential Equations first
#
# An ordinary differential equation (ODE) is an equation involving functions having one variable.
#
@@ -3206,8 +3206,8 @@ def sigmoid(z):
# but with number of hidden layers specified by the user.
def deep_neural_network(deep_params, x):
# N_hidden is the number of hidden layers
N_hidden = np.size(deep_params) - 1 # -1 since params consists of
# deep_params is a list, len() should be used
N_hidden = len(deep_params) - 1 # -1 since params consists of
# parameters to all the hidden
# layers AND the output layer.
@@ -3436,9 +3436,12 @@ def get_parameters():
g0 = 1.2
return alpha, A, g0
def deep_neural_network(P, x):
def deep_neural_network(deep_params, x):
# N_hidden is the number of hidden layers
N_hidden = np.size(P) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
# deep_params is a list, len() should be used
N_hidden = len(deep_params) - 1 # -1 since params consists of
# parameters to all the hidden
# layers AND the output layer.
# Assumes input x being an one-dimensional array
num_values = np.size(x)
@@ -3455,7 +3458,7 @@ def deep_neural_network(P, x):
for l in range(N_hidden):
# From the list of parameters P; find the correct weigths and bias for this layer
w_hidden = P[l]
w_hidden = deep_params[l]
# Add a row of ones to include bias
x_prev = np.concatenate((np.ones((1,num_values)), x_prev ), axis = 0)
@@ -3469,7 +3472,7 @@ def deep_neural_network(P, x):
## Output layer:
# Get the weights and bias for this layer
w_output = P[-1]
w_output = deep_params[-1]
# Include bias:
x_prev = np.concatenate((np.ones((1,num_values)), x_prev), axis = 0)
@@ -3480,6 +3483,8 @@ def deep_neural_network(P, x):
return x_output
def cost_function_deep(P, x):
# Evaluate the trial function with the current parameters P
@@ -3785,7 +3790,10 @@ def sigmoid(z):
def deep_neural_network(deep_params, x):
# N_hidden is the number of hidden layers
N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
# deep_params is a list, len() should be used
N_hidden = len(deep_params) - 1 # -1 since params consists of
# parameters to all the hidden
# layers AND the output layer.
# Assumes input x being an one-dimensional array
num_values = np.size(x)
@@ -3826,6 +3834,7 @@ def deep_neural_network(deep_params, x):
return x_output
def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):
# num_hidden_neurons is now a list of number of neurons within each hidden layer
@@ -4035,7 +4044,10 @@ def sigmoid(z):
def deep_neural_network(deep_params, x):
# N_hidden is the number of hidden layers
N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
# deep_params is a list, len() should be used
N_hidden = len(deep_params) - 1 # -1 since params consists of
# parameters to all the hidden
# layers AND the output layer.
# Assumes input x being an one-dimensional array
num_values = np.size(x)
@@ -4076,6 +4088,7 @@ def deep_neural_network(deep_params, x):
return x_output
def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):
# num_hidden_neurons is now a list of number of neurons within each hidden layer
@@ -4358,7 +4371,7 @@ def deep_neural_network(deep_params, x):
num_points = np.size(x,1)
# N_hidden is the number of hidden layers
N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
N_hidden = len(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
# Assume that the input layer does nothing to the input x
x_input = x
@@ -4516,7 +4529,7 @@ def deep_neural_network(deep_params, x):
num_points = np.size(x,1)
# N_hidden is the number of hidden layers
N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
N_hidden = len(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
# Assume that the input layer does nothing to the input x
x_input = x
@@ -4659,7 +4672,7 @@ if __name__ == '__main__':
T,X = np.meshgrid(t,x)
fig = plt.figure(figsize=(10,10))
ax = fig.gca(projection='3d')
ax = fig.add_suplot(projection='3d')
ax.set_title('Solution from the deep neural network w/ %d layer'%len(num_hidden_neurons))
s = ax.plot_surface(T,X,g_dnn_ag,linewidth=0,antialiased=False,cmap=cm.viridis)
ax.set_xlabel('Time $t$')
@@ -4667,14 +4680,14 @@ if __name__ == '__main__':
fig = plt.figure(figsize=(10,10))
ax = fig.gca(projection='3d')
ax = fig.add_suplot(projection='3d')
ax.set_title('Analytical solution')
s = ax.plot_surface(T,X,G_analytical,linewidth=0,antialiased=False,cmap=cm.viridis)
ax.set_xlabel('Time $t$')
ax.set_ylabel('Position $x$');
fig = plt.figure(figsize=(10,10))
ax = fig.gca(projection='3d')
ax = fig.add_suplot(projection='3d')
ax.set_title('Difference')
s = ax.plot_surface(T,X,diff_ag,linewidth=0,antialiased=False,cmap=cm.viridis)
ax.set_xlabel('Time $t$')
@@ -4859,7 +4872,7 @@ def deep_neural_network(deep_params, x):
num_points = np.size(x,1)
# N_hidden is the number of hidden layers
N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
N_hidden = len(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer
# Assume that the input layer does nothing to the input x
x_input = x
@@ -4964,7 +4977,7 @@ if __name__ == '__main__':
T,X = np.meshgrid(t,x)
fig = plt.figure(figsize=(10,10))
ax = fig.gca(projection='3d')
ax = fig.add_suplot(projection='3d')
ax.set_title('Solution from the deep neural network w/ %d layer'%len(num_hidden_neurons))
s = ax.plot_surface(T,X,res,linewidth=0,antialiased=False,cmap=cm.viridis)
ax.set_xlabel('Time $t$')
@@ -4972,7 +4985,7 @@ if __name__ == '__main__':
fig = plt.figure(figsize=(10,10))
ax = fig.gca(projection='3d')
ax = fig.add_suplot(projection='3d')
ax.set_title('Analytical solution')
s = ax.plot_surface(T,X,res_analytical,linewidth=0,antialiased=False,cmap=cm.viridis)
ax.set_xlabel('Time $t$')
@@ -4980,7 +4993,7 @@ if __name__ == '__main__':
fig = plt.figure(figsize=(10,10))
ax = fig.gca(projection='3d')
ax = fig.add_suplot(projection='3d')
ax.set_title('Difference')
s = ax.plot_surface(T,X,diff,linewidth=0,antialiased=False,cmap=cm.viridis)
ax.set_xlabel('Time $t$')
doc/LectureNotes/week43.ipynb
+1329 -662
View File
File diff suppressed because it is too large Load Diff
doc/pub/week43/html/week43-bs.html
+2 -2
View File
@@ -488,8 +488,8 @@ MathJax.Hub.Config({
<!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
<ul>
<li> Building our own Feed-forward Neural Network with intro to Tensorflow</li>
<li> Solving differential equations with Neural Networks
<!-- * <a href="https://youtu.be/" target="_self">Video of lecture to be posted asap</a> -->
<li> Solving differential equations with Neural Networks</li>
<li> "Video of lecture at <a href="https://youtu.be/vkBNTn-MLqs" target="_self"><tt>https://youtu.be/vkBNTn-MLqs</tt></a>
<!-- * <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOct21.pdf" target="_self">Whiteboard notes</a> --></li>
</ul>
</div>
doc/pub/week43/html/week43-reveal.html
+3 -2
View File
@@ -204,8 +204,9 @@ MathJax.Hub.Config({
<p><li> Building our own Feed-forward Neural Network with intro to Tensorflow</li>
<p><li> Solving differential equations with Neural Networks
<!-- * <a href="https://youtu.be/" target="_blank">Video of lecture to be posted asap</a> -->
<p><li> Solving differential equations with Neural Networks</li>
<p><li> "Video of lecture at <a href="https://youtu.be/vkBNTn-MLqs" target="_blank"><tt>https://youtu.be/vkBNTn-MLqs</tt></a>
<!-- * <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOct21.pdf" target="_blank">Whiteboard notes</a> --></li>
</ul>
</div>
doc/pub/week43/html/week43-solarized.html
+2 -2
View File
@@ -384,8 +384,8 @@ MathJax.Hub.Config({
<p>
<ul>
<li> Building our own Feed-forward Neural Network with intro to Tensorflow</li>
<li> Solving differential equations with Neural Networks
<!-- * <a href="https://youtu.be/" target="_blank">Video of lecture to be posted asap</a> -->
<li> Solving differential equations with Neural Networks</li>
<li> "Video of lecture at <a href="https://youtu.be/vkBNTn-MLqs" target="_blank"><tt>https://youtu.be/vkBNTn-MLqs</tt></a>
<!-- * <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOct21.pdf" target="_blank">Whiteboard notes</a> --></li>
</ul>
</div>
doc/pub/week43/html/week43.html
+2 -2
View File
@@ -461,8 +461,8 @@ MathJax.Hub.Config({
<p>
<ul>
<li> Building our own Feed-forward Neural Network with intro to Tensorflow</li>
<li> Solving differential equations with Neural Networks
<!-- * <a href="https://youtu.be/" target="_blank">Video of lecture to be posted asap</a> -->
<li> Solving differential equations with Neural Networks</li>
<li> "Video of lecture at <a href="https://youtu.be/vkBNTn-MLqs" target="_blank"><tt>https://youtu.be/vkBNTn-MLqs</tt></a>
<!-- * <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOct21.pdf" target="_blank">Whiteboard notes</a> --></li>
</ul>
</div>
doc/pub/week43/ipynb/ipynb-week43-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week43/ipynb/week43.ipynb
+313 -312
View File
File diff suppressed because it is too large Load Diff
doc/src/week43/week43.do.txt
+1 -1
View File
@@ -8,7 +8,7 @@ DATE: October 21, 2024
!bblock Material for the lecture on Monday October 21, 2024
* Building our own Feed-forward Neural Network with intro to Tensorflow
* Solving differential equations with Neural Networks
# * "Video of lecture to be posted asap":"https://youtu.be/"
* "Video of lecture at URL:"https://youtu.be/vkBNTn-MLqs"
# * "Whiteboard notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOct21.pdf"
!eblock