lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update book

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-10-21 13:17:43 +02:00
parent d307e15b25
commit 0a03ba583c
7 changed files with 644 additions and 1004 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/week43.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/_sources/week43.ipynb
+313 -312
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/html/week43.html
+12 -139
View File
@@ -1510,8 +1510,8 @@ doconce format html week43.do.txt --no_mako -->
<ul class="simple">
<li><p>Building our own Feed-forward Neural Network with intro to Tensorflow</p></li>
<li><p>Solving differential equations with Neural Networks</p></li>
<li><p>“Video of lecture at <a class="reference external" href="https://youtu.be/vkBNTn-MLqs">https://youtu.be/vkBNTn-MLqs</a></p></li>
</ul>
<!-- * [Video of lecture to be posted asap](https://youtu.be/) -->
<!-- * [Whiteboard notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOct21.pdf) --></div>
<div class="section" id="exercises-and-lab-session-week-43">
<h2>Exercises and lab session week 43<a class="headerlink" href="#exercises-and-lab-session-week-43" title="Permalink to this headline"></a></h2>
@@ -2346,7 +2346,7 @@ the <em>Hadamard product</em>, meaning element-wise multiplication.</p>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Old accuracy on training data: 0.1440501043841336
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_93488/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2677,134 +2677,6 @@ Lambda = 1.0
Accuracy score on test set: 0.7694444444444445
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
Lambda = 10.0
Accuracy score on test set: 0.19166666666666668
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 1e-05
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 0.0001
Accuracy score on test set: 0.08611111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 0.001
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 0.01
Accuracy score on test set: 0.08888888888888889
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 0.1
Accuracy score on test set: 0.08611111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 1.0
Accuracy score on test set: 0.08888888888888889
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 10.0
Accuracy score on test set: 0.09166666666666666
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 1e-05
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 0.0001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 0.001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 0.01
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_92604/1630775253.py:44: RuntimeWarning: invalid value encountered in divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">KeyboardInterrupt</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="n">Cell</span> <span class="n">In</span><span class="p">[</span><span class="mi">8</span><span class="p">],</span> <span class="n">line</span> <span class="mi">11</span>
@@ -2815,17 +2687,18 @@ Accuracy score on test set: 0.07777777777777778
<span class="g g-Whitespace"> </span><span class="mi">13</span> <span class="n">DNN_numpy</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">dnn</span>
<span class="g g-Whitespace"> </span><span class="mi">15</span> <span class="n">test_predict</span> <span class="o">=</span> <span class="n">dnn</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="nn">Cell In[6], line 99,</span> in <span class="ni">NeuralNetwork.train</span><span class="nt">(self)</span>
<span class="nn">Cell In[6], line 98,</span> in <span class="ni">NeuralNetwork.train</span><span class="nt">(self)</span>
<span class="g g-Whitespace"> </span><span class="mi">95</span> <span class="bp">self</span><span class="o">.</span><span class="n">X_data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">X_data_full</span><span class="p">[</span><span class="n">chosen_datapoints</span><span class="p">]</span>
<span class="g g-Whitespace"> </span><span class="mi">96</span> <span class="bp">self</span><span class="o">.</span><span class="n">Y_data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">Y_data_full</span><span class="p">[</span><span class="n">chosen_datapoints</span><span class="p">]</span>
<span class="g g-Whitespace"> </span><span class="mi">98</span> <span class="bp">self</span><span class="o">.</span><span class="n">feed_forward</span><span class="p">()</span>
<span class="ne">---&gt; </span><span class="mi">99</span> <span class="bp">self</span><span class="o">.</span><span class="n">backpropagation</span><span class="p">()</span>
<span class="ne">---&gt; </span><span class="mi">98</span> <span class="bp">self</span><span class="o">.</span><span class="n">feed_forward</span><span class="p">()</span>
<span class="g g-Whitespace"> </span><span class="mi">99</span> <span class="bp">self</span><span class="o">.</span><span class="n">backpropagation</span><span class="p">()</span>
<span class="nn">Cell In[6], line 64,</span> in <span class="ni">NeuralNetwork.backpropagation</span><span class="nt">(self)</span>
<span class="g g-Whitespace"> </span><span class="mi">61</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_weights_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">a_h</span><span class="o">.</span><span class="n">T</span><span class="p">,</span> <span class="n">error_output</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">62</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_bias_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">error_output</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="ne">---&gt; </span><span class="mi">64</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_weights_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">X_data</span><span class="o">.</span><span class="n">T</span><span class="p">,</span> <span class="n">error_hidden</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">65</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_bias_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">error_hidden</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">67</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">lmbd</span> <span class="o">&gt;</span> <span class="mf">0.0</span><span class="p">:</span>
<span class="nn">Cell In[6], line 38,</span> in <span class="ni">NeuralNetwork.feed_forward</span><span class="nt">(self)</span>
<span class="g g-Whitespace"> </span><span class="mi">36</span> <span class="k">def</span> <span class="nf">feed_forward</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="g g-Whitespace"> </span><span class="mi">37</span> <span class="c1"># feed-forward for training</span>
<span class="ne">---&gt; </span><span class="mi">38</span> <span class="bp">self</span><span class="o">.</span><span class="n">z_h</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">X_data</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_weights</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_bias</span>
<span class="g g-Whitespace"> </span><span class="mi">39</span> <span class="bp">self</span><span class="o">.</span><span class="n">a_h</span> <span class="o">=</span> <span class="n">sigmoid</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z_h</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">41</span> <span class="bp">self</span><span class="o">.</span><span class="n">z_o</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">a_h</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_weights</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_bias</span>
<span class="ne">KeyboardInterrupt</span>:
</pre></div>
doc/LectureNotes/_build/jupyter_execute/week43.ipynb
+316 -551
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/jupyter_execute/week43.py
+2 -1
View File
@@ -17,7 +17,8 @@
# * Building our own Feed-forward Neural Network with intro to Tensorflow
#
# * Solving differential equations with Neural Networks
# <!-- * [Video of lecture to be posted asap](https://youtu.be/) -->
#
# * "Video of lecture at <https://youtu.be/vkBNTn-MLqs>
# <!-- * [Whiteboard notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOct21.pdf) -->
# ## Exercises and lab session week 43