update

doc/LectureNotes/_build/html/_sources/exercisesweek41.ipynb

@@ -3,9 +3,7 @@
   {
    "cell_type": "markdown",
    "id": "b18bcd06",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
     "doconce format html exercisesweek41.do.txt  -->\n",
@@ -15,9 +13,7 @@
   {
    "cell_type": "markdown",
    "id": "7542d6aa",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "# Exercises week 41\n",
     "**October 4-11, 2024**\n",
@@ -28,9 +24,7 @@
   {
    "cell_type": "markdown",
    "id": "80943a15",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "# Overarching aims of the exercises this week\n",
     "\n",
@@ -83,9 +77,7 @@
   {
    "cell_type": "markdown",
    "id": "2095d197",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "# Code examples from week 39 and 40"
    ]
@@ -93,9 +85,7 @@
   {
    "cell_type": "markdown",
    "id": "f428decb",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Code with a Number of Minibatches which varies, analytical gradient\n",
     "\n",
@@ -106,10 +96,7 @@
    "cell_type": "code",
    "execution_count": 1,
    "id": "ba38d454",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "%matplotlib inline\n",
@@ -185,9 +172,7 @@
   {
    "cell_type": "markdown",
    "id": "de04b41a",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "In the above code, we have use replacement in setting up the\n",
     "mini-batches. The discussion\n",
@@ -198,9 +183,7 @@
   {
    "cell_type": "markdown",
    "id": "77fc1cca",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Momentum based GD\n",
     "\n",
@@ -213,9 +196,7 @@
   {
    "cell_type": "markdown",
    "id": "441d1f36",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n",
@@ -225,9 +206,7 @@
   {
    "cell_type": "markdown",
    "id": "47434945",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "<!-- Equation labels as ordinary links -->\n",
     "<div id=\"_auto1\"></div>\n",
@@ -243,9 +222,7 @@
   {
    "cell_type": "markdown",
    "id": "f3ea5060",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "where we have introduced a momentum parameter $\\gamma$, with\n",
     "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n",
@@ -262,9 +239,7 @@
   {
    "cell_type": "markdown",
    "id": "923628c8",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n",
@@ -274,9 +249,7 @@
   {
    "cell_type": "markdown",
    "id": "5c94031c",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$."
    ]
@@ -284,9 +257,7 @@
   {
    "cell_type": "markdown",
    "id": "f3f0e9c9",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Algorithms and codes for Adagrad, RMSprop and Adam\n",
     "\n",
@@ -298,9 +269,7 @@
   {
    "cell_type": "markdown",
    "id": "92253eff",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Practical tips\n",
     "\n",
@@ -318,9 +287,7 @@
   {
    "cell_type": "markdown",
    "id": "08209015",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Using Automatic differentation with OLS\n",
     "\n",
@@ -333,10 +300,7 @@
    "cell_type": "code",
    "execution_count": 2,
    "id": "f1f7d4aa",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Using Autograd to calculate gradients for OLS\n",
@@ -393,9 +357,7 @@
   {
    "cell_type": "markdown",
    "id": "1bc83f33",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Same code but now with momentum gradient descent"
    ]
@@ -404,10 +366,7 @@
    "cell_type": "code",
    "execution_count": 3,
    "id": "dc2a3f65",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Using Autograd to calculate gradients for OLS\n",
@@ -468,9 +427,7 @@
   {
    "cell_type": "markdown",
    "id": "0ef007d0",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## But noen of these can compete with Newton's method"
    ]
@@ -479,10 +436,7 @@
    "cell_type": "code",
    "execution_count": 4,
    "id": "0e498aa4",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Using Newton's method\n",
@@ -528,9 +482,7 @@
   {
    "cell_type": "markdown",
    "id": "40292cf3",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Including Stochastic Gradient Descent with Autograd\n",
     "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**."
@@ -540,10 +492,7 @@
    "cell_type": "code",
    "execution_count": 5,
    "id": "fa819b9d",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Using Autograd to calculate gradients using SGD\n",
@@ -624,9 +573,7 @@
   {
    "cell_type": "markdown",
    "id": "2ca466b4",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Same code but now with momentum gradient descent"
    ]
@@ -635,10 +582,7 @@
    "cell_type": "code",
    "execution_count": 6,
    "id": "0d44a49c",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Using Autograd to calculate gradients using SGD\n",
@@ -713,9 +657,7 @@
   {
    "cell_type": "markdown",
    "id": "b82627f6",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## AdaGrad algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
     "\n",
@@ -729,9 +671,7 @@
   {
    "cell_type": "markdown",
    "id": "00d3aff0",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Similar (second order function now) problem but now with AdaGrad"
    ]
@@ -740,10 +680,7 @@
    "cell_type": "code",
    "execution_count": 7,
    "id": "6b85aacc",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n",
@@ -799,9 +736,7 @@
   {
    "cell_type": "markdown",
    "id": "d8ddde38",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Running this code we note an almost perfect agreement with the results from matrix inversion."
    ]
@@ -809,9 +744,7 @@
   {
    "cell_type": "markdown",
    "id": "ff15b503",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## RMSProp algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
     "\n",
@@ -825,9 +758,7 @@
   {
    "cell_type": "markdown",
    "id": "66f96d12",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## RMSprop for adaptive learning rate with Stochastic Gradient Descent"
    ]
@@ -836,10 +767,7 @@
    "cell_type": "code",
    "execution_count": 8,
    "id": "888f1b4e",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Using Autograd to calculate gradients using RMSprop  and Stochastic Gradient descent\n",
@@ -901,9 +829,7 @@
   {
    "cell_type": "markdown",
    "id": "2e0860f7",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## ADAM algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
     "\n",
@@ -917,9 +843,7 @@
   {
    "cell_type": "markdown",
    "id": "ab4a9859",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## And finally [ADAM](https://arxiv.org/pdf/1412.6980.pdf)"
    ]
@@ -928,10 +852,7 @@
    "cell_type": "code",
    "execution_count": 9,
    "id": "ccdd4d77",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Using Autograd to calculate gradients using RMSprop  and Stochastic Gradient descent\n",
@@ -998,9 +919,7 @@
   {
    "cell_type": "markdown",
    "id": "25ac988c",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n",
     "\n",
@@ -1014,9 +933,7 @@
   {
    "cell_type": "markdown",
    "id": "37d556d0",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Getting started with Jax, note the way we import numpy"
    ]
@@ -1025,10 +942,7 @@
    "cell_type": "code",
    "execution_count": 10,
    "id": "5b81d6e4",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "import jax\n",
@@ -1042,9 +956,7 @@
   {
    "cell_type": "markdown",
    "id": "c42db672",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### A warm-up example"
    ]
@@ -1053,10 +965,7 @@
    "cell_type": "code",
    "execution_count": 11,
    "id": "98eb2f26",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "def function(x):\n",
@@ -1098,9 +1007,7 @@
   {
    "cell_type": "markdown",
    "id": "8a5f19b5",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### A more advanced example"
    ]
@@ -1109,10 +1016,7 @@
    "cell_type": "code",
    "execution_count": 12,
    "id": "d8f5eb38",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "backend = np\n",
@@ -1138,7 +1042,25 @@
    ]
   }
  ],
- "metadata": {},
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  }
+ },
  "nbformat": 4,
  "nbformat_minor": 5
 }

doc/LectureNotes/_build/html/_sources/project2.ipynb

@@ -3,9 +3,7 @@
   {
    "cell_type": "markdown",
    "id": "5b2f9dda",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
     "doconce format html Project2.do.txt  -->\n",
@@ -15,9 +13,7 @@
   {
    "cell_type": "markdown",
    "id": "cacbd604",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "# Project 2 on Machine Learning, deadline November 4 (Midnight)\n",
     "**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, Department of Physics, University of Oslo, Norway\n",
@@ -30,9 +26,7 @@
   {
    "cell_type": "markdown",
    "id": "acb32119",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Classification and Regression, from linear and logistic regression to neural networks\n",
     "\n",
@@ -76,9 +70,7 @@
   {
    "cell_type": "markdown",
    "id": "027202f0",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Part a): Write your own Stochastic Gradient Descent  code, first step\n",
     "\n",
@@ -132,9 +124,7 @@
   {
    "cell_type": "markdown",
    "id": "9388fa74",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Part b): Writing your own Neural Network code\n",
     "\n",
@@ -170,9 +160,7 @@
   {
    "cell_type": "markdown",
    "id": "49666354",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Part c): Testing different activation functions\n",
     "\n",
@@ -182,9 +170,7 @@
   {
    "cell_type": "markdown",
    "id": "79aacf29",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Part d): Classification  analysis using neural networks\n",
     "\n",
@@ -208,9 +194,7 @@
   {
    "cell_type": "markdown",
    "id": "42e22900",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\text{Accuracy} = \\frac{\\sum_{i=1}^n I(t_i = y_i)}{n} ,\n",
@@ -220,9 +204,7 @@
   {
    "cell_type": "markdown",
    "id": "82ae763d",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$\n",
     "otherwise if we have a binary classification problem. Here $t_i$\n",
@@ -240,9 +222,7 @@
   {
    "cell_type": "markdown",
    "id": "1d6b84d1",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Part e): Write your Logistic Regression code, final step\n",
     "\n",
@@ -262,9 +242,7 @@
   {
    "cell_type": "markdown",
    "id": "0bce8832",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Part f) Critical evaluation of the various algorithms\n",
     "\n",
@@ -278,9 +256,7 @@
   {
    "cell_type": "markdown",
    "id": "51b1b29b",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Background literature\n",
     "\n",
@@ -294,9 +270,7 @@
   {
    "cell_type": "markdown",
    "id": "7e4ffbbd",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Introduction to numerical projects\n",
     "\n",
@@ -325,9 +299,7 @@
   {
    "cell_type": "markdown",
    "id": "56112b03",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Format for electronic delivery of report and programs\n",
     "\n",
@@ -345,7 +317,25 @@
    ]
   }
  ],
- "metadata": {},
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  }
+ },
  "nbformat": 4,
  "nbformat_minor": 5
 }

doc/LectureNotes/_build/html/chapter1.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -1036,13 +1056,13 @@ example of the functionality of <strong>Scikit-Learn</strong>.</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The intercept alpha: 
- [1.92622842]
+ [1.94485679]
 Coefficient beta : 
- [[5.21332621]]
-Mean squared error: 0.22
-Variance score: 0.90
+ [[5.13059483]]
+Mean squared error: 0.32
+Variance score: 0.87
 Mean squared log error: 0.01
-Mean absolute error: 0.38
+Mean absolute error: 0.45
 </pre></div>
 </div>
 <img alt="_images/chapter1_19_1.png" src="_images/chapter1_19_1.png" />
@@ -1142,7 +1162,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
 </div>
 <div class="cell_output docutils container">
 <img alt="_images/chapter1_33_0.png" src="_images/chapter1_33_0.png" />
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.005
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.004999999999999996
 </pre></div>
 </div>
 </div>

doc/LectureNotes/_build/html/chapter10.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -1353,7 +1373,7 @@ the <em>Hadamard product</em>, meaning element-wise multiplication.</p>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Old accuracy on training data: 0.1440501043841336
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1687,7 +1707,7 @@ Lambda =  10.0
 Accuracy score on test set:  0.19166666666666668
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1696,7 +1716,7 @@ Lambda =  1e-05
 Accuracy score on test set:  0.10555555555555556
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1705,7 +1725,7 @@ Lambda =  0.0001
 Accuracy score on test set:  0.08611111111111111
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1714,7 +1734,7 @@ Lambda =  0.001
 Accuracy score on test set:  0.10555555555555556
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1723,7 +1743,7 @@ Lambda =  0.01
 Accuracy score on test set:  0.08888888888888889
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1732,7 +1752,7 @@ Lambda =  0.1
 Accuracy score on test set:  0.08611111111111111
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1741,7 +1761,7 @@ Lambda =  1.0
 Accuracy score on test set:  0.08888888888888889
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1750,11 +1770,11 @@ Lambda =  10.0
 Accuracy score on test set:  0.09166666666666666
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
   exp_term = np.exp(self.z_o)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
   self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1763,11 +1783,11 @@ Lambda =  1e-05
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
   exp_term = np.exp(self.z_o)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
   self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1776,11 +1796,11 @@ Lambda =  0.0001
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
   exp_term = np.exp(self.z_o)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
   self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1789,11 +1809,11 @@ Lambda =  0.001
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
   exp_term = np.exp(self.z_o)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
   self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1802,11 +1822,11 @@ Lambda =  0.01
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
   exp_term = np.exp(self.z_o)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
   self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1815,7 +1835,7 @@ Lambda =  0.1
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1824,11 +1844,11 @@ Lambda =  1.0
 Accuracy score on test set:  0.10555555555555556
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
   exp_term = np.exp(self.z_o)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
   self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1837,11 +1857,11 @@ Lambda =  10.0
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
   exp_term = np.exp(self.z_o)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
   self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1850,37 +1870,82 @@ Lambda =  1e-05
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
   return 1/(1 + np.exp(-x))
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
   exp_term = np.exp(self.z_o)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95354/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
   self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
-<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
-<span class="ne">KeyboardInterrupt</span><span class="g g-Whitespace">                         </span>Traceback (most recent call last)
-<span class="n">Cell</span> <span class="n">In</span><span class="p">[</span><span class="mi">8</span><span class="p">],</span> <span class="n">line</span> <span class="mi">11</span>
-<span class="g g-Whitespace">      </span><span class="mi">8</span> <span class="k">for</span> <span class="n">j</span><span class="p">,</span> <span class="n">lmbd</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">lmbd_vals</span><span class="p">):</span>
-<span class="g g-Whitespace">      </span><span class="mi">9</span>     <span class="n">dnn</span> <span class="o">=</span> <span class="n">NeuralNetwork</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">Y_train_onehot</span><span class="p">,</span> <span class="n">eta</span><span class="o">=</span><span class="n">eta</span><span class="p">,</span> <span class="n">lmbd</span><span class="o">=</span><span class="n">lmbd</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="n">epochs</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span>
-<span class="g g-Whitespace">     </span><span class="mi">10</span>                         <span class="n">n_hidden_neurons</span><span class="o">=</span><span class="n">n_hidden_neurons</span><span class="p">,</span> <span class="n">n_categories</span><span class="o">=</span><span class="n">n_categories</span><span class="p">)</span>
-<span class="ne">---&gt; </span><span class="mi">11</span>     <span class="n">dnn</span><span class="o">.</span><span class="n">train</span><span class="p">()</span>
-<span class="g g-Whitespace">     </span><span class="mi">13</span>     <span class="n">DNN_numpy</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">dnn</span>
-<span class="g g-Whitespace">     </span><span class="mi">15</span>     <span class="n">test_predict</span> <span class="o">=</span> <span class="n">dnn</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
-
-<span class="nn">Cell In[6], line 99,</span> in <span class="ni">NeuralNetwork.train</span><span class="nt">(self)</span>
-<span class="g g-Whitespace">     </span><span class="mi">96</span> <span class="bp">self</span><span class="o">.</span><span class="n">Y_data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">Y_data_full</span><span class="p">[</span><span class="n">chosen_datapoints</span><span class="p">]</span>
-<span class="g g-Whitespace">     </span><span class="mi">98</span> <span class="bp">self</span><span class="o">.</span><span class="n">feed_forward</span><span class="p">()</span>
-<span class="ne">---&gt; </span><span class="mi">99</span> <span class="bp">self</span><span class="o">.</span><span class="n">backpropagation</span><span class="p">()</span>
-
-<span class="nn">Cell In[6], line 64,</span> in <span class="ni">NeuralNetwork.backpropagation</span><span class="nt">(self)</span>
-<span class="g g-Whitespace">     </span><span class="mi">61</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_weights_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">a_h</span><span class="o">.</span><span class="n">T</span><span class="p">,</span> <span class="n">error_output</span><span class="p">)</span>
-<span class="g g-Whitespace">     </span><span class="mi">62</span> <span class="bp">self</span><span class="o">.</span><span class="n">output_bias_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">error_output</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
-<span class="ne">---&gt; </span><span class="mi">64</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_weights_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">X_data</span><span class="o">.</span><span class="n">T</span><span class="p">,</span> <span class="n">error_hidden</span><span class="p">)</span>
-<span class="g g-Whitespace">     </span><span class="mi">65</span> <span class="bp">self</span><span class="o">.</span><span class="n">hidden_bias_gradient</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">error_hidden</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
-<span class="g g-Whitespace">     </span><span class="mi">67</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">lmbd</span> <span class="o">&gt;</span> <span class="mf">0.0</span><span class="p">:</span>
-
-<span class="ne">KeyboardInterrupt</span>: 
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  0.0001
+Accuracy score on test set:  0.07777777777777778
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+  exp_term = np.exp(self.z_o)
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  0.001
+Accuracy score on test set:  0.07777777777777778
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+  exp_term = np.exp(self.z_o)
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  0.01
+Accuracy score on test set:  0.07777777777777778
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+  exp_term = np.exp(self.z_o)
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  0.1
+Accuracy score on test set:  0.07777777777777778
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+  exp_term = np.exp(self.z_o)
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  1.0
+Accuracy score on test set:  0.07777777777777778
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+  exp_term = np.exp(self.z_o)
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  10.0
+Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
 </div>
@@ -1926,6 +1991,22 @@ Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22411/953065564.py:4: RuntimeWarning: overflow encountered in exp
+  return 1/(1 + np.exp(-x))
+</pre></div>
+</div>
+<img alt="_images/chapter10_59_1.png" src="_images/chapter10_59_1.png" />
+<img alt="_images/chapter10_59_2.png" src="_images/chapter10_59_2.png" />
+</div>
 </div>
 </div>
 <div class="section" id="scikit-learn-implementation">
@@ -1961,6 +2042,333 @@ performance overall.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1e-05
+Lambda =  1e-05
+Accuracy score on test set:  0.18333333333333332
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1e-05
+Lambda =  0.0001
+Accuracy score on test set:  0.18611111111111112
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1e-05
+Lambda =  0.001
+Accuracy score on test set:  0.13055555555555556
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1e-05
+Lambda =  0.01
+Accuracy score on test set:  0.24444444444444444
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1e-05
+Lambda =  0.1
+Accuracy score on test set:  0.23333333333333334
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1e-05
+Lambda =  1.0
+Accuracy score on test set:  0.12777777777777777
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1e-05
+Lambda =  10.0
+Accuracy score on test set:  0.1527777777777778
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.0001
+Lambda =  1e-05
+Accuracy score on test set:  0.9111111111111111
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.0001
+Lambda =  0.0001
+Accuracy score on test set:  0.8888888888888888
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.0001
+Lambda =  0.001
+Accuracy score on test set:  0.8722222222222222
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.0001
+Lambda =  0.01
+Accuracy score on test set:  0.8305555555555556
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.0001
+Lambda =  0.1
+Accuracy score on test set:  0.8888888888888888
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.0001
+Lambda =  1.0
+Accuracy score on test set:  0.8805555555555555
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.0001
+Lambda =  10.0
+Accuracy score on test set:  0.8944444444444445
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.001
+Lambda =  1e-05
+Accuracy score on test set:  0.975
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.001
+Lambda =  0.0001
+Accuracy score on test set:  0.9777777777777777
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.001
+Lambda =  0.001
+Accuracy score on test set:  0.9805555555555555
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.001
+Lambda =  0.01
+Accuracy score on test set:  0.9861111111111112
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.001
+Lambda =  0.1
+Accuracy score on test set:  0.9805555555555555
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.001
+Lambda =  1.0
+Accuracy score on test set:  0.9777777777777777
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.001
+Lambda =  10.0
+Accuracy score on test set:  0.9444444444444444
+</pre></div>
+</div>
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
+  warnings.warn(
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.01
+Lambda =  1e-05
+Accuracy score on test set:  0.9861111111111112
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.01
+Lambda =  0.0001
+Accuracy score on test set:  0.9888888888888889
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.01
+Lambda =  0.001
+Accuracy score on test set:  0.9888888888888889
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.01
+Lambda =  0.01
+Accuracy score on test set:  0.9861111111111112
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.01
+Lambda =  0.1
+Accuracy score on test set:  0.9888888888888889
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.01
+Lambda =  1.0
+Accuracy score on test set:  0.9722222222222222
+
+Learning rate  =  0.01
+Lambda =  10.0
+Accuracy score on test set:  0.9527777777777777
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.1
+Lambda =  1e-05
+Accuracy score on test set:  0.9027777777777778
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.1
+Lambda =  0.0001
+Accuracy score on test set:  0.8583333333333333
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.1
+Lambda =  0.001
+Accuracy score on test set:  0.8722222222222222
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.1
+Lambda =  0.01
+Accuracy score on test set:  0.9055555555555556
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.1
+Lambda =  0.1
+Accuracy score on test set:  0.8805555555555555
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.1
+Lambda =  1.0
+Accuracy score on test set:  0.8722222222222222
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  0.1
+Lambda =  10.0
+Accuracy score on test set:  0.8666666666666667
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1.0
+Lambda =  1e-05
+Accuracy score on test set:  0.08611111111111111
+
+Learning rate  =  1.0
+Lambda =  0.0001
+Accuracy score on test set:  0.10555555555555556
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1.0
+Lambda =  0.001
+Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  1.0
+Lambda =  0.01
+Accuracy score on test set:  0.17777777777777778
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1.0
+Lambda =  0.1
+Accuracy score on test set:  0.08333333333333333
+
+Learning rate  =  1.0
+Lambda =  1.0
+Accuracy score on test set:  0.08888888888888889
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  1.0
+Lambda =  10.0
+Accuracy score on test set:  0.09444444444444444
+
+Learning rate  =  10.0
+Lambda =  1e-05
+Accuracy score on test set:  0.17222222222222222
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  0.0001
+Accuracy score on test set:  0.11666666666666667
+
+Learning rate  =  10.0
+Lambda =  0.001
+Accuracy score on test set:  0.10555555555555556
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  0.01
+Accuracy score on test set:  0.1388888888888889
+
+Learning rate  =  10.0
+Lambda =  0.1
+Accuracy score on test set:  0.11388888888888889
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
+Lambda =  1.0
+Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  10.0
+Lambda =  10.0
+Accuracy score on test set:  0.09444444444444444
+</pre></div>
+</div>
+</div>
 </div>
 </div>
 <div class="section" id="id1">
@@ -2004,6 +2412,10 @@ performance overall.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<img alt="_images/chapter10_63_0.png" src="_images/chapter10_63_0.png" />
+<img alt="_images/chapter10_63_1.png" src="_images/chapter10_63_1.png" />
+</div>
 </div>
 </div>
 <div class="section" id="building-neural-networks-in-tensorflow-and-keras">
@@ -2042,6 +2454,14 @@ and/or if you use <strong>anaconda</strong>, just write (or install from the gra
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span>  <span class="n">Cell</span> <span class="n">In</span><span class="p">[</span><span class="mi">12</span><span class="p">],</span> <span class="n">line</span> <span class="mi">1</span>
+    <span class="n">conda</span> <span class="n">create</span> <span class="o">-</span><span class="n">n</span> <span class="n">tf</span> <span class="n">tensorflow</span>
+          <span class="o">^</span>
+<span class="ne">SyntaxError</span>: invalid syntax
+</pre></div>
+</div>
+</div>
 </div>
 <p>To install the current release of GPU TensorFlow</p>
 <div class="cell docutils container">

doc/LectureNotes/_build/html/chapter11.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -2587,11 +2607,83 @@ Using TensorFlow results in a much better execution time. Try it!</p>
 <span class="g g-Whitespace">     </span><span class="mi">19</span>     <span class="n">x</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">argnum</span><span class="p">)</span>
 <span class="ne">---&gt; </span><span class="mi">20</span> <span class="k">return</span> <span class="n">unary_operator</span><span class="p">(</span><span class="n">unary_f</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="o">*</span><span class="n">nary_op_args</span><span class="p">,</span> <span class="o">**</span><span class="n">nary_op_kwargs</span><span class="p">)</span>
 
-<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/differential_operators.py:32,</span> in <span class="ni">grad</span><span class="nt">(fun, x)</span>
-<span class="g g-Whitespace">     </span><span class="mi">29</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">vspace</span><span class="p">(</span><span class="n">ans</span><span class="p">)</span><span class="o">.</span><span class="n">size</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
-<span class="g g-Whitespace">     </span><span class="mi">30</span>     <span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">&quot;Grad only applies to real scalar-output functions. &quot;</span>
-<span class="g g-Whitespace">     </span><span class="mi">31</span>                     <span class="s2">&quot;Try jacobian, elementwise_grad or holomorphic_grad.&quot;</span><span class="p">)</span>
-<span class="ne">---&gt; </span><span class="mi">32</span> <span class="k">return</span> <span class="n">vjp</span><span class="p">(</span><span class="n">vspace</span><span class="p">(</span><span class="n">ans</span><span class="p">)</span><span class="o">.</span><span class="n">ones</span><span class="p">())</span>
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/differential_operators.py:28,</span> in <span class="ni">grad</span><span class="nt">(fun, x)</span>
+<span class="g g-Whitespace">     </span><span class="mi">21</span> <span class="nd">@unary_to_nary</span>
+<span class="g g-Whitespace">     </span><span class="mi">22</span> <span class="k">def</span> <span class="nf">grad</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
+<span class="g g-Whitespace">     </span><span class="mi">23</span><span class="w">     </span><span class="sd">&quot;&quot;&quot;</span>
+<span class="g g-Whitespace">     </span><span class="mi">24</span><span class="sd">     Returns a function which computes the gradient of `fun` with respect to</span>
+<span class="g g-Whitespace">     </span><span class="mi">25</span><span class="sd">     positional argument number `argnum`. The returned function takes the same</span>
+<span class="g g-Whitespace">     </span><span class="mi">26</span><span class="sd">     arguments as `fun`, but returns the gradient instead. The function `fun`</span>
+<span class="g g-Whitespace">     </span><span class="mi">27</span><span class="sd">     should be scalar-valued. The gradient has the same type as the argument.&quot;&quot;&quot;</span>
+<span class="ne">---&gt; </span><span class="mi">28</span>     <span class="n">vjp</span><span class="p">,</span> <span class="n">ans</span> <span class="o">=</span> <span class="n">_make_vjp</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">29</span>     <span class="k">if</span> <span class="ow">not</span> <span class="n">vspace</span><span class="p">(</span><span class="n">ans</span><span class="p">)</span><span class="o">.</span><span class="n">size</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">30</span>         <span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">&quot;Grad only applies to real scalar-output functions. &quot;</span>
+<span class="g g-Whitespace">     </span><span class="mi">31</span>                         <span class="s2">&quot;Try jacobian, elementwise_grad or holomorphic_grad.&quot;</span><span class="p">)</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/core.py:10,</span> in <span class="ni">make_vjp</span><span class="nt">(fun, x)</span>
+<span class="g g-Whitespace">      </span><span class="mi">8</span> <span class="k">def</span> <span class="nf">make_vjp</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
+<span class="g g-Whitespace">      </span><span class="mi">9</span>     <span class="n">start_node</span> <span class="o">=</span> <span class="n">VJPNode</span><span class="o">.</span><span class="n">new_root</span><span class="p">()</span>
+<span class="ne">---&gt; </span><span class="mi">10</span>     <span class="n">end_value</span><span class="p">,</span> <span class="n">end_node</span> <span class="o">=</span>  <span class="n">trace</span><span class="p">(</span><span class="n">start_node</span><span class="p">,</span> <span class="n">fun</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">11</span>     <span class="k">if</span> <span class="n">end_node</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">12</span>         <span class="k">def</span> <span class="nf">vjp</span><span class="p">(</span><span class="n">g</span><span class="p">):</span> <span class="k">return</span> <span class="n">vspace</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">zeros</span><span class="p">()</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/tracer.py:10,</span> in <span class="ni">trace</span><span class="nt">(start_node, fun, x)</span>
+<span class="g g-Whitespace">      </span><span class="mi">8</span> <span class="k">with</span> <span class="n">trace_stack</span><span class="o">.</span><span class="n">new_trace</span><span class="p">()</span> <span class="k">as</span> <span class="n">t</span><span class="p">:</span>
+<span class="g g-Whitespace">      </span><span class="mi">9</span>     <span class="n">start_box</span> <span class="o">=</span> <span class="n">new_box</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">t</span><span class="p">,</span> <span class="n">start_node</span><span class="p">)</span>
+<span class="ne">---&gt; </span><span class="mi">10</span>     <span class="n">end_box</span> <span class="o">=</span> <span class="n">fun</span><span class="p">(</span><span class="n">start_box</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">11</span>     <span class="k">if</span> <span class="n">isbox</span><span class="p">(</span><span class="n">end_box</span><span class="p">)</span> <span class="ow">and</span> <span class="n">end_box</span><span class="o">.</span><span class="n">_trace</span> <span class="o">==</span> <span class="n">start_box</span><span class="o">.</span><span class="n">_trace</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">12</span>         <span class="k">return</span> <span class="n">end_box</span><span class="o">.</span><span class="n">_value</span><span class="p">,</span> <span class="n">end_box</span><span class="o">.</span><span class="n">_node</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/wrap_util.py:15,</span> in <span class="ni">unary_to_nary.&lt;locals&gt;.nary_operator.&lt;locals&gt;.nary_f.&lt;locals&gt;.unary_f</span><span class="nt">(x)</span>
+<span class="g g-Whitespace">     </span><span class="mi">13</span> <span class="k">else</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">14</span>     <span class="n">subargs</span> <span class="o">=</span> <span class="n">subvals</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="nb">zip</span><span class="p">(</span><span class="n">argnum</span><span class="p">,</span> <span class="n">x</span><span class="p">))</span>
+<span class="ne">---&gt; </span><span class="mi">15</span> <span class="k">return</span> <span class="n">fun</span><span class="p">(</span><span class="o">*</span><span class="n">subargs</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+<span class="nn">Cell In[9], line 80,</span> in <span class="ni">cost_function</span><span class="nt">(P, x, t)</span>
+<span class="g g-Whitespace">     </span><span class="mi">78</span> <span class="n">g_t</span> <span class="o">=</span> <span class="n">g_trial</span><span class="p">(</span><span class="n">point</span><span class="p">,</span><span class="n">P</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">79</span> <span class="n">g_t_jacobian</span> <span class="o">=</span> <span class="n">g_t_jacobian_func</span><span class="p">(</span><span class="n">point</span><span class="p">,</span><span class="n">P</span><span class="p">)</span>
+<span class="ne">---&gt; </span><span class="mi">80</span> <span class="n">g_t_hessian</span> <span class="o">=</span> <span class="n">g_t_hessian_func</span><span class="p">(</span><span class="n">point</span><span class="p">,</span><span class="n">P</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">82</span> <span class="n">g_t_dt</span> <span class="o">=</span> <span class="n">g_t_jacobian</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
+<span class="g g-Whitespace">     </span><span class="mi">83</span> <span class="n">g_t_d2x</span> <span class="o">=</span> <span class="n">g_t_hessian</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/wrap_util.py:20,</span> in <span class="ni">unary_to_nary.&lt;locals&gt;.nary_operator.&lt;locals&gt;.nary_f</span><span class="nt">(*args, **kwargs)</span>
+<span class="g g-Whitespace">     </span><span class="mi">18</span> <span class="k">else</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">19</span>     <span class="n">x</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">argnum</span><span class="p">)</span>
+<span class="ne">---&gt; </span><span class="mi">20</span> <span class="k">return</span> <span class="n">unary_operator</span><span class="p">(</span><span class="n">unary_f</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="o">*</span><span class="n">nary_op_args</span><span class="p">,</span> <span class="o">**</span><span class="n">nary_op_kwargs</span><span class="p">)</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/differential_operators.py:81,</span> in <span class="ni">hessian</span><span class="nt">(fun, x)</span>
+<span class="g g-Whitespace">     </span><span class="mi">78</span> <span class="nd">@unary_to_nary</span>
+<span class="g g-Whitespace">     </span><span class="mi">79</span> <span class="k">def</span> <span class="nf">hessian</span><span class="p">(</span><span class="n">fun</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
+<span class="g g-Whitespace">     </span><span class="mi">80</span>     <span class="s2">&quot;Returns a function that computes the exact Hessian.&quot;</span>
+<span class="ne">---&gt; </span><span class="mi">81</span>     <span class="k">return</span> <span class="n">jacobian</span><span class="p">(</span><span class="n">jacobian</span><span class="p">(</span><span class="n">fun</span><span class="p">))(</span><span class="n">x</span><span class="p">)</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/wrap_util.py:20,</span> in <span class="ni">unary_to_nary.&lt;locals&gt;.nary_operator.&lt;locals&gt;.nary_f</span><span class="nt">(*args, **kwargs)</span>
+<span class="g g-Whitespace">     </span><span class="mi">18</span> <span class="k">else</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">19</span>     <span class="n">x</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">argnum</span><span class="p">)</span>
+<span class="ne">---&gt; </span><span class="mi">20</span> <span class="k">return</span> <span class="n">unary_operator</span><span class="p">(</span><span class="n">unary_f</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="o">*</span><span class="n">nary_op_args</span><span class="p">,</span> <span class="o">**</span><span class="n">nary_op_kwargs</span><span class="p">)</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/differential_operators.py:64,</span> in <span class="ni">jacobian</span><span class="nt">(fun, x)</span>
+<span class="g g-Whitespace">     </span><span class="mi">62</span> <span class="n">jacobian_shape</span> <span class="o">=</span> <span class="n">ans_vspace</span><span class="o">.</span><span class="n">shape</span> <span class="o">+</span> <span class="n">vspace</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">shape</span>
+<span class="g g-Whitespace">     </span><span class="mi">63</span> <span class="n">grads</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="n">vjp</span><span class="p">,</span> <span class="n">ans_vspace</span><span class="o">.</span><span class="n">standard_basis</span><span class="p">())</span>
+<span class="ne">---&gt; </span><span class="mi">64</span> <span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">stack</span><span class="p">(</span><span class="n">grads</span><span class="p">),</span> <span class="n">jacobian_shape</span><span class="p">)</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/numpy/numpy_wrapper.py:88,</span> in <span class="ni">stack</span><span class="nt">(arrays, axis)</span>
+<span class="g g-Whitespace">     </span><span class="mi">83</span> <span class="k">def</span> <span class="nf">stack</span><span class="p">(</span><span class="n">arrays</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span>
+<span class="g g-Whitespace">     </span><span class="mi">84</span>     <span class="c1"># this code is basically copied from numpy/core/shape_base.py&#39;s stack</span>
+<span class="g g-Whitespace">     </span><span class="mi">85</span>     <span class="c1"># we need it here because we want to re-implement stack in terms of the</span>
+<span class="g g-Whitespace">     </span><span class="mi">86</span>     <span class="c1"># primitives defined in this file</span>
+<span class="ne">---&gt; </span><span class="mi">88</span>     <span class="n">arrays</span> <span class="o">=</span> <span class="p">[</span><span class="n">array</span><span class="p">(</span><span class="n">arr</span><span class="p">)</span> <span class="k">for</span> <span class="n">arr</span> <span class="ow">in</span> <span class="n">arrays</span><span class="p">]</span>
+<span class="g g-Whitespace">     </span><span class="mi">89</span>     <span class="k">if</span> <span class="ow">not</span> <span class="n">arrays</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">90</span>         <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">&#39;need at least one array to stack&#39;</span><span class="p">)</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/numpy/numpy_wrapper.py:88,</span> in <span class="ni">&lt;listcomp&gt;</span><span class="nt">(.0)</span>
+<span class="g g-Whitespace">     </span><span class="mi">83</span> <span class="k">def</span> <span class="nf">stack</span><span class="p">(</span><span class="n">arrays</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span>
+<span class="g g-Whitespace">     </span><span class="mi">84</span>     <span class="c1"># this code is basically copied from numpy/core/shape_base.py&#39;s stack</span>
+<span class="g g-Whitespace">     </span><span class="mi">85</span>     <span class="c1"># we need it here because we want to re-implement stack in terms of the</span>
+<span class="g g-Whitespace">     </span><span class="mi">86</span>     <span class="c1"># primitives defined in this file</span>
+<span class="ne">---&gt; </span><span class="mi">88</span>     <span class="n">arrays</span> <span class="o">=</span> <span class="p">[</span><span class="n">array</span><span class="p">(</span><span class="n">arr</span><span class="p">)</span> <span class="k">for</span> <span class="n">arr</span> <span class="ow">in</span> <span class="n">arrays</span><span class="p">]</span>
+<span class="g g-Whitespace">     </span><span class="mi">89</span>     <span class="k">if</span> <span class="ow">not</span> <span class="n">arrays</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">90</span>         <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">&#39;need at least one array to stack&#39;</span><span class="p">)</span>
 
 <span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/core.py:14,</span> in <span class="ni">make_vjp.&lt;locals&gt;.vjp</span><span class="nt">(g)</span>
 <span class="ne">---&gt; </span><span class="mi">14</span> <span class="k">def</span> <span class="nf">vjp</span><span class="p">(</span><span class="n">g</span><span class="p">):</span> <span class="k">return</span> <span class="n">backward_pass</span><span class="p">(</span><span class="n">g</span><span class="p">,</span> <span class="n">end_node</span><span class="p">)</span>
@@ -2603,44 +2695,44 @@ Using TensorFlow results in a much better execution time. Try it!</p>
 <span class="g g-Whitespace">     </span><span class="mi">22</span>     <span class="k">for</span> <span class="n">parent</span><span class="p">,</span> <span class="n">ingrad</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">parents</span><span class="p">,</span> <span class="n">ingrads</span><span class="p">):</span>
 <span class="g g-Whitespace">     </span><span class="mi">23</span>         <span class="n">outgrads</span><span class="p">[</span><span class="n">parent</span><span class="p">]</span> <span class="o">=</span> <span class="n">add_outgrads</span><span class="p">(</span><span class="n">outgrads</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">parent</span><span class="p">),</span> <span class="n">ingrad</span><span class="p">)</span>
 
-<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/core.py:67,</span> in <span class="ni">defvjp.&lt;locals&gt;.vjp_argnums.&lt;locals&gt;.&lt;lambda&gt;</span><span class="nt">(g)</span>
-<span class="g g-Whitespace">     </span><span class="mi">64</span>         <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span>
-<span class="g g-Whitespace">     </span><span class="mi">65</span>             <span class="s2">&quot;VJP of </span><span class="si">{}</span><span class="s2"> wrt argnum 0 not defined&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">fun</span><span class="o">.</span><span class="vm">__name__</span><span class="p">))</span>
-<span class="g g-Whitespace">     </span><span class="mi">66</span>     <span class="n">vjp</span> <span class="o">=</span> <span class="n">vjpfun</span><span class="p">(</span><span class="n">ans</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
-<span class="ne">---&gt; </span><span class="mi">67</span>     <span class="k">return</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="p">(</span><span class="n">vjp</span><span class="p">(</span><span class="n">g</span><span class="p">),)</span>
-<span class="g g-Whitespace">     </span><span class="mi">68</span> <span class="k">elif</span> <span class="n">L</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span>
-<span class="g g-Whitespace">     </span><span class="mi">69</span>     <span class="n">argnum_0</span><span class="p">,</span> <span class="n">argnum_1</span> <span class="o">=</span> <span class="n">argnums</span>
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/core.py:78,</span> in <span class="ni">defvjp.&lt;locals&gt;.vjp_argnums.&lt;locals&gt;.&lt;lambda&gt;</span><span class="nt">(g)</span>
+<span class="g g-Whitespace">     </span><span class="mi">76</span>     <span class="n">vjp_0</span> <span class="o">=</span> <span class="n">vjp_0_fun</span><span class="p">(</span><span class="n">ans</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">77</span>     <span class="n">vjp_1</span> <span class="o">=</span> <span class="n">vjp_1_fun</span><span class="p">(</span><span class="n">ans</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+<span class="ne">---&gt; </span><span class="mi">78</span>     <span class="k">return</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="p">(</span><span class="n">vjp_0</span><span class="p">(</span><span class="n">g</span><span class="p">),</span> <span class="n">vjp_1</span><span class="p">(</span><span class="n">g</span><span class="p">))</span>
+<span class="g g-Whitespace">     </span><span class="mi">79</span> <span class="k">else</span><span class="p">:</span>
+<span class="g g-Whitespace">     </span><span class="mi">80</span>     <span class="n">vjps</span> <span class="o">=</span> <span class="p">[</span><span class="n">vjps_dict</span><span class="p">[</span><span class="n">argnum</span><span class="p">](</span><span class="n">ans</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">for</span> <span class="n">argnum</span> <span class="ow">in</span> <span class="n">argnums</span><span class="p">]</span>
 
 <span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/numpy/numpy_vjps.py:660,</span> in <span class="ni">unbroadcast_f.&lt;locals&gt;.&lt;lambda&gt;</span><span class="nt">(g)</span>
 <span class="g g-Whitespace">    </span><span class="mi">658</span> <span class="k">def</span> <span class="nf">unbroadcast_f</span><span class="p">(</span><span class="n">target</span><span class="p">,</span> <span class="n">f</span><span class="p">):</span>
 <span class="g g-Whitespace">    </span><span class="mi">659</span>     <span class="n">target_meta</span> <span class="o">=</span> <span class="n">anp</span><span class="o">.</span><span class="n">metadata</span><span class="p">(</span><span class="n">target</span><span class="p">)</span>
 <span class="ne">--&gt; </span><span class="mi">660</span>     <span class="k">return</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="n">unbroadcast</span><span class="p">(</span><span class="n">f</span><span class="p">(</span><span class="n">g</span><span class="p">),</span> <span class="n">target_meta</span><span class="p">)</span>
 
-<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/numpy/numpy_vjps.py:653,</span> in <span class="ni">unbroadcast</span><span class="nt">(x, target_meta, broadcast_idx)</span>
-<span class="g g-Whitespace">    </span><span class="mi">651</span> <span class="k">for</span> <span class="n">axis</span><span class="p">,</span> <span class="n">size</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">target_shape</span><span class="p">):</span>
-<span class="g g-Whitespace">    </span><span class="mi">652</span>     <span class="k">if</span> <span class="n">size</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
-<span class="ne">--&gt; </span><span class="mi">653</span>         <span class="n">x</span> <span class="o">=</span> <span class="n">anp</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="n">axis</span><span class="p">,</span> <span class="n">keepdims</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
-<span class="g g-Whitespace">    </span><span class="mi">654</span> <span class="k">if</span> <span class="n">anp</span><span class="o">.</span><span class="n">iscomplexobj</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">target_iscomplex</span><span class="p">:</span>
-<span class="g g-Whitespace">    </span><span class="mi">655</span>     <span class="n">x</span> <span class="o">=</span> <span class="n">anp</span><span class="o">.</span><span class="n">real</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/numpy/numpy_vjps.py:34,</span> in <span class="ni">&lt;lambda&gt;</span><span class="nt">(g)</span>
+<span class="g g-Whitespace">     </span><span class="mi">30</span> <span class="c1"># ----- Binary ufuncs -----</span>
+<span class="g g-Whitespace">     </span><span class="mi">32</span> <span class="n">defvjp</span><span class="p">(</span><span class="n">anp</span><span class="o">.</span><span class="n">add</span><span class="p">,</span>         <span class="k">lambda</span> <span class="n">ans</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">:</span> <span class="n">unbroadcast_f</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="n">g</span><span class="p">),</span>
+<span class="g g-Whitespace">     </span><span class="mi">33</span>                         <span class="k">lambda</span> <span class="n">ans</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">:</span> <span class="n">unbroadcast_f</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="n">g</span><span class="p">))</span>
+<span class="ne">---&gt; </span><span class="mi">34</span> <span class="n">defvjp</span><span class="p">(</span><span class="n">anp</span><span class="o">.</span><span class="n">multiply</span><span class="p">,</span>    <span class="k">lambda</span> <span class="n">ans</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">:</span> <span class="n">unbroadcast_f</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="n">y</span> <span class="o">*</span> <span class="n">g</span><span class="p">),</span>
+<span class="g g-Whitespace">     </span><span class="mi">35</span>                         <span class="k">lambda</span> <span class="n">ans</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">:</span> <span class="n">unbroadcast_f</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="n">x</span> <span class="o">*</span> <span class="n">g</span><span class="p">))</span>
+<span class="g g-Whitespace">     </span><span class="mi">36</span> <span class="n">defvjp</span><span class="p">(</span><span class="n">anp</span><span class="o">.</span><span class="n">subtract</span><span class="p">,</span>    <span class="k">lambda</span> <span class="n">ans</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">:</span> <span class="n">unbroadcast_f</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="n">g</span><span class="p">),</span>
+<span class="g g-Whitespace">     </span><span class="mi">37</span>                         <span class="k">lambda</span> <span class="n">ans</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">:</span> <span class="n">unbroadcast_f</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="o">-</span><span class="n">g</span><span class="p">))</span>
+<span class="g g-Whitespace">     </span><span class="mi">38</span> <span class="n">defvjp</span><span class="p">(</span><span class="n">anp</span><span class="o">.</span><span class="n">divide</span><span class="p">,</span>      <span class="k">lambda</span> <span class="n">ans</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">:</span> <span class="n">unbroadcast_f</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span>   <span class="n">g</span> <span class="o">/</span> <span class="n">y</span><span class="p">),</span>
+<span class="g g-Whitespace">     </span><span class="mi">39</span>                         <span class="k">lambda</span> <span class="n">ans</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">:</span> <span class="n">unbroadcast_f</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">g</span><span class="p">:</span> <span class="o">-</span> <span class="n">g</span> <span class="o">*</span> <span class="n">x</span> <span class="o">/</span> <span class="n">y</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/numpy/numpy_boxes.py:27,</span> in <span class="ni">ArrayBox.__mul__</span><span class="nt">(self, other)</span>
+<span class="ne">---&gt; </span><span class="mi">27</span> <span class="k">def</span> <span class="fm">__mul__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="k">return</span> <span class="n">anp</span><span class="o">.</span><span class="n">multiply</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">)</span>
+
+<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/tracer.py:44,</span> in <span class="ni">primitive.&lt;locals&gt;.f_wrapped</span><span class="nt">(*args, **kwargs)</span>
+<span class="g g-Whitespace">     </span><span class="mi">42</span> <span class="n">parents</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">box</span><span class="o">.</span><span class="n">_node</span> <span class="k">for</span> <span class="n">_</span>     <span class="p">,</span> <span class="n">box</span> <span class="ow">in</span> <span class="n">boxed_args</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">43</span> <span class="n">argnums</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">argnum</span>    <span class="k">for</span> <span class="n">argnum</span><span class="p">,</span> <span class="n">_</span>   <span class="ow">in</span> <span class="n">boxed_args</span><span class="p">)</span>
+<span class="ne">---&gt; </span><span class="mi">44</span> <span class="n">ans</span> <span class="o">=</span> <span class="n">f_wrapped</span><span class="p">(</span><span class="o">*</span><span class="n">argvals</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">45</span> <span class="n">node</span> <span class="o">=</span> <span class="n">node_constructor</span><span class="p">(</span><span class="n">ans</span><span class="p">,</span> <span class="n">f_wrapped</span><span class="p">,</span> <span class="n">argvals</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">,</span> <span class="n">argnums</span><span class="p">,</span> <span class="n">parents</span><span class="p">)</span>
+<span class="g g-Whitespace">     </span><span class="mi">46</span> <span class="k">return</span> <span class="n">new_box</span><span class="p">(</span><span class="n">ans</span><span class="p">,</span> <span class="n">trace</span><span class="p">,</span> <span class="n">node</span><span class="p">)</span>
 
 <span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/autograd/tracer.py:48,</span> in <span class="ni">primitive.&lt;locals&gt;.f_wrapped</span><span class="nt">(*args, **kwargs)</span>
 <span class="g g-Whitespace">     </span><span class="mi">46</span>     <span class="k">return</span> <span class="n">new_box</span><span class="p">(</span><span class="n">ans</span><span class="p">,</span> <span class="n">trace</span><span class="p">,</span> <span class="n">node</span><span class="p">)</span>
 <span class="g g-Whitespace">     </span><span class="mi">47</span> <span class="k">else</span><span class="p">:</span>
 <span class="ne">---&gt; </span><span class="mi">48</span>     <span class="k">return</span> <span class="n">f_raw</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
 
-<span class="nn">File &lt;__array_function__ internals&gt;:180,</span> in <span class="ni">sum</span><span class="nt">(*args, **kwargs)</span>
-
-<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/numpy/core/fromnumeric.py:2296,</span> in <span class="ni">sum</span><span class="nt">(a, axis, dtype, out, keepdims, initial, where)</span>
-<span class="g g-Whitespace">   </span><span class="mi">2293</span>         <span class="k">return</span> <span class="n">out</span>
-<span class="g g-Whitespace">   </span><span class="mi">2294</span>     <span class="k">return</span> <span class="n">res</span>
-<span class="ne">-&gt; </span><span class="mi">2296</span> <span class="k">return</span> <span class="n">_wrapreduction</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">add</span><span class="p">,</span> <span class="s1">&#39;sum&#39;</span><span class="p">,</span> <span class="n">axis</span><span class="p">,</span> <span class="n">dtype</span><span class="p">,</span> <span class="n">out</span><span class="p">,</span> <span class="n">keepdims</span><span class="o">=</span><span class="n">keepdims</span><span class="p">,</span>
-<span class="g g-Whitespace">   </span><span class="mi">2297</span>                       <span class="n">initial</span><span class="o">=</span><span class="n">initial</span><span class="p">,</span> <span class="n">where</span><span class="o">=</span><span class="n">where</span><span class="p">)</span>
-
-<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/numpy/core/fromnumeric.py:86,</span> in <span class="ni">_wrapreduction</span><span class="nt">(obj, ufunc, method, axis, dtype, out, **kwargs)</span>
-<span class="g g-Whitespace">     </span><span class="mi">83</span>         <span class="k">else</span><span class="p">:</span>
-<span class="g g-Whitespace">     </span><span class="mi">84</span>             <span class="k">return</span> <span class="n">reduction</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="n">axis</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">out</span><span class="p">,</span> <span class="o">**</span><span class="n">passkwargs</span><span class="p">)</span>
-<span class="ne">---&gt; </span><span class="mi">86</span> <span class="k">return</span> <span class="n">ufunc</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="n">axis</span><span class="p">,</span> <span class="n">dtype</span><span class="p">,</span> <span class="n">out</span><span class="p">,</span> <span class="o">**</span><span class="n">passkwargs</span><span class="p">)</span>
-
 <span class="ne">KeyboardInterrupt</span>: 
 </pre></div>
 </div>

doc/LectureNotes/_build/html/chapter12.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/chapter13.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/chapter2.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -1285,10 +1305,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.024792624800382218
-3.9225384545636204
-[[0.94623184 2.84401886]
- [2.84401886 9.4477214 ]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.009134699065945493
+4.0244965108017645
+[[0.85613835 2.50655379]
+ [2.50655379 8.3404509 ]]
 </pre></div>
 </div>
 </div>
@@ -1325,10 +1345,10 @@ a more brute force way. Here we scale the mean values for each column of the des
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08536248571780691
-1.8207274870895702
-[[1.         0.74900488]
- [0.74900488 1.        ]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07971187802560528
+1.800782161095708
+[[1.         0.59411814]
+ [0.59411814 1.        ]]
 </pre></div>
 </div>
 </div>
@@ -1358,32 +1378,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 1.10115017  1.66431407]
- [ 0.12043521  1.32305911]
- [-1.30023144 -3.36154104]
- [-0.25200841 -1.11277166]
- [-1.55102329 -4.20158083]
- [ 0.72770687  0.97206657]
- [ 0.76533281  2.10747579]
- [-0.20666447  0.79623487]
- [-0.63919355 -2.48490503]
- [ 1.23449607  4.29764815]]
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>          0         1
-0  1.101150  1.664314
-1  0.120435  1.323059
-2 -1.300231 -3.361541
-3 -0.252008 -1.112772
-4 -1.551023 -4.201581
-5  0.727707  0.972067
-6  0.765333  2.107476
-7 -0.206664  0.796235
-8 -0.639194 -2.484905
-9  1.234496  4.297648
-        0       1
-0  1.0000  0.9434
-1  0.9434  1.0000
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 0.81395716  1.89155934]
+ [-1.34726166 -4.13453411]
+ [-0.46229544 -2.34061974]
+ [ 0.24429334  1.4051634 ]
+ [ 0.41971814  1.6405671 ]
+ [ 2.02456235  5.03973227]
+ [-1.97311824 -4.72521196]
+ [ 0.10738656  0.24578123]
+ [-0.52702419 -2.34023682]
+ [ 0.69978197  3.31779928]]
+          0         1
+0  0.813957  1.891559
+1 -1.347262 -4.134534
+2 -0.462295 -2.340620
+3  0.244293  1.405163
+4  0.419718  1.640567
+5  2.024562  5.039732
+6 -1.973118 -4.725212
+7  0.107387  0.245781
+8 -0.527024 -2.340237
+9  0.699782  3.317799
+          0         1
+0  1.000000  0.969413
+1  0.969413  1.000000
 </pre></div>
 </div>
 </div>
@@ -1440,37 +1458,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>     0         1         2         3         4         5         6         7   \
 0   0.0  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   
-1   0.0  0.083179  0.086483  0.081217  0.083548  0.086239  0.072037  0.073514   
-2   0.0  0.086483  0.091362  0.082533  0.085853  0.089626  0.071638  0.073800   
-3   0.0  0.081217  0.082533  0.084963  0.086068  0.087429  0.079030  0.079670   
-4   0.0  0.083548  0.085853  0.086068  0.087871  0.089996  0.078945  0.080094   
-5   0.0  0.086239  0.089626  0.087429  0.089996  0.092956  0.079002  0.080706   
-6   0.0  0.072037  0.071638  0.079030  0.078945  0.079002  0.076071  0.075876   
-7   0.0  0.073514  0.073800  0.079670  0.080094  0.080706  0.075876  0.076066   
-8   0.0  0.075289  0.076321  0.080549  0.081527  0.082742  0.075842  0.076448   
-9   0.0  0.077391  0.079240  0.081686  0.083268  0.085145  0.075978  0.077035   
-10  0.0  0.063786  0.062216  0.072380  0.071437  0.070552  0.071436  0.070622   
-11  0.0  0.064606  0.063536  0.072591  0.072031  0.071560  0.071049  0.070531   
-12  0.0  0.065654  0.065121  0.072997  0.072849  0.072826  0.070805  0.070605   
-13  0.0  0.066948  0.067000  0.073612  0.073911  0.074374  0.070712  0.070855   
-14  0.0  0.068512  0.069203  0.074454  0.075239  0.076232  0.070780  0.071294   
+1   0.0  0.092746  0.090302  0.090561  0.088058  0.085463  0.081530  0.078898   
+2   0.0  0.090302  0.088694  0.089052  0.086797  0.084423  0.080286  0.077782   
+3   0.0  0.090561  0.089052  0.095106  0.092533  0.089858  0.089404  0.086489   
+4   0.0  0.088058  0.086797  0.092533  0.090114  0.087585  0.086913  0.084127   
+5   0.0  0.085463  0.084423  0.089858  0.087585  0.085197  0.084340  0.081681   
+6   0.0  0.081530  0.080286  0.089404  0.086913  0.084340  0.086471  0.083596   
+7   0.0  0.078898  0.077782  0.086489  0.084127  0.081681  0.083596  0.080849   
+8   0.0  0.076334  0.075334  0.083645  0.081405  0.079080  0.080793  0.078170   
+9   0.0  0.073841  0.072946  0.080875  0.078753  0.076543  0.078067  0.075562   
+10  0.0  0.072973  0.071789  0.082361  0.079982  0.077541  0.081296  0.078544   
+11  0.0  0.070485  0.069391  0.079518  0.077254  0.074928  0.078454  0.075823   
+12  0.0  0.068088  0.067077  0.076777  0.074622  0.072404  0.075712  0.073198   
+13  0.0  0.065778  0.064845  0.074134  0.072083  0.069970  0.073071  0.070667   
+14  0.0  0.063554  0.062693  0.071588  0.069637  0.067623  0.070527  0.068229   
 
           8         9         10        11        12        13        14  
 0   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  
-1   0.075289  0.077391  0.063786  0.064606  0.065654  0.066948  0.068512  
-2   0.076321  0.079240  0.062216  0.063536  0.065121  0.067000  0.069203  
-3   0.080549  0.081686  0.072380  0.072591  0.072997  0.073612  0.074454  
-4   0.081527  0.083268  0.071437  0.072031  0.072849  0.073911  0.075239  
-5   0.082742  0.085145  0.070552  0.071560  0.072826  0.074374  0.076232  
-6   0.075842  0.075978  0.071436  0.071049  0.070805  0.070712  0.070780  
-7   0.076448  0.077035  0.070622  0.070531  0.070605  0.070855  0.071294  
-8   0.077280  0.078359  0.069907  0.070135  0.070552  0.071173  0.072015  
-9   0.078359  0.079976  0.069293  0.069866  0.070655  0.071680  0.072961  
-10  0.069907  0.069293  0.068353  0.067515  0.066778  0.066145  0.065619  
-11  0.070135  0.069866  0.067515  0.066912  0.066425  0.066059  0.065821  
-12  0.070552  0.070655  0.066778  0.066425  0.066205  0.066127  0.066199  
-13  0.071173  0.071680  0.066145  0.066059  0.066127  0.066358  0.066766  
-14  0.072015  0.072961  0.065619  0.065821  0.066199  0.066766  0.067539  
+1   0.076334  0.073841  0.072973  0.070485  0.068088  0.065778  0.063554  
+2   0.075334  0.072946  0.071789  0.069391  0.067077  0.064845  0.062693  
+3   0.083645  0.080875  0.082361  0.079518  0.076777  0.074134  0.071588  
+4   0.081405  0.078753  0.079982  0.077254  0.074622  0.072083  0.069637  
+5   0.079080  0.076543  0.077541  0.074928  0.072404  0.069970  0.067623  
+6   0.080793  0.078067  0.081296  0.078454  0.075712  0.073071  0.070527  
+7   0.078170  0.075562  0.078544  0.075823  0.073198  0.070667  0.068229  
+8   0.075609  0.073115  0.075864  0.073260  0.070747  0.068324  0.065989  
+9   0.073115  0.070730  0.073260  0.070769  0.068364  0.066044  0.063809  
+10  0.075864  0.073260  0.077608  0.074866  0.072223  0.069676  0.067224  
+11  0.073260  0.070769  0.074866  0.072241  0.069711  0.067273  0.064924  
+12  0.070747  0.068364  0.072223  0.069711  0.067288  0.064954  0.062705  
+13  0.068324  0.066044  0.069676  0.067273  0.064954  0.062719  0.060565  
+14  0.065989  0.063809  0.067224  0.064924  0.062705  0.060565  0.058503  
 </pre></div>
 </div>
 </div>

doc/LectureNotes/_build/html/chapter3.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -839,10 +859,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.147545 sec
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.146141 sec
 Jackknife Statistics :
 original           bias      std. error
-  99.977         99.967        0.152494
+ 100.139        100.129        0.148776
 </pre></div>
 </div>
 </div>
@@ -1061,7 +1081,7 @@ theorem.</p>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Bootstrap Statistics :
 original           bias      std. error
- 100.041  14.8133        100.041        0.149266
+  99.989  15.1792        99.9878        0.152149
 </pre></div>
 </div>
 </div>
@@ -1278,9 +1298,7 @@ Error: 0.06547790180152355
 Bias^2: 0.06208238634231949
 Var: 0.0033955154592040936
 0.06547790180152355 &gt;= 0.06208238634231949 + 0.0033955154592040936 = 0.06547790180152359
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 4
+Polynomial degree: 4
 Error: 0.06844519414009445
 Bias^2: 0.06453579006728324
 Var: 0.003909404072811226
@@ -1307,21 +1325,19 @@ Error: 0.017355848195593347
 Bias^2: 0.010331721306655127
 Var: 0.007024126888938232
 0.017355848195593347 &gt;= 0.010331721306655127 + 0.007024126888938232 = 0.01735584819559336
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 9
+Polynomial degree: 9
 Error: 0.02660572763718093
 Bias^2: 0.010018312644137363
 Var: 0.016587414993043573
 0.02660572763718093 &gt;= 0.010018312644137363 + 0.016587414993043573 = 0.026605727637180936
-Polynomial degree: 10
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 10
 Error: 0.021592704588025025
 Bias^2: 0.010516485576645508
 Var: 0.011076219011379514
 0.021592704588025025 &gt;= 0.010516485576645508 + 0.011076219011379514 = 0.021592704588025022
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 11
+Polynomial degree: 11
 Error: 0.07160048164233104
 Bias^2: 0.014436800088904942
 Var: 0.05716368155342608
@@ -1331,14 +1347,16 @@ Error: 0.11547777218872497
 Bias^2: 0.01628578269596628
 Var: 0.09919198949275869
 0.11547777218872497 &gt;= 0.01628578269596628 + 0.09919198949275869 = 0.11547777218872497
-Polynomial degree: 13
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 13
 Error: 0.22842468702219465
 Bias^2: 0.01975416527185249
 Var: 0.20867052175034223
 0.22842468702219465 &gt;= 0.01975416527185249 + 0.20867052175034223 = 0.2284246870221947
 </pre></div>
 </div>
-<img alt="_images/chapter3_66_5.png" src="_images/chapter3_66_5.png" />
+<img alt="_images/chapter3_66_4.png" src="_images/chapter3_66_4.png" />
 </div>
 </div>
 <p>The bias-variance tradeoff summarizes the fundamental tension in
@@ -1653,9 +1671,9 @@ Mean squared error on training data: 0.00060704
 Mean squared error on test data: 3262.26814548
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95419/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22458/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
   plt.plot(polynomial, np.log10(trainingerror), label=&#39;Training Error&#39;)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95419/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22458/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
   plt.plot(polynomial, np.log10(testerror), label=&#39;Test Error&#39;)
 </pre></div>
 </div>
@@ -1889,7 +1907,7 @@ cross-validation (LOOCV).</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95419/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22458/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
   plt.plot(polynomial, np.log10(estimated_mse_sklearn), label=&#39;Test Error&#39;)
 </pre></div>
 </div>
@@ -2778,7 +2796,7 @@ linear system as an equation would reduce this down to
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95419/4162706317.py:7: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22458/4162706317.py:7: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
   cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
 </pre></div>
 </div>
@@ -2922,7 +2940,7 @@ with the form utilized in linear regression, viz.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95419/3777801602.py:7: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22458/3777801602.py:7: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
   cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
 </pre></div>
 </div>
@@ -2962,7 +2980,7 @@ cost function is given by</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95419/438060758.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22458/438060758.py:10: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
   cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
 </pre></div>
 </div>
@@ -2997,7 +3015,7 @@ cost function is given by</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95419/3544313922.py:9: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22458/3544313922.py:9: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
   cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
 </pre></div>
 </div>
@@ -3050,43 +3068,43 @@ constant as opposed to ridge and OLS. We get a sparse solution with
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>  0%|                                                                                                                        | 0/10 [00:00&lt;?, ?it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>  0%|                                                                                                                                                                                                        | 0/10 [00:00&lt;?, ?it/s]
 </pre></div>
 </div>
 <div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_coordinate_descent.py:628: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.924e+00, tolerance: 1.797e+00
   model = cd_fast.enet_coordinate_descent(
 
- 10%|███████████▏                                                                                                    | 1/10 [00:00&lt;00:06,  1.35it/s]
+ 10%|███████████████████▏                                                                                                                                                                            | 1/10 [00:00&lt;00:06,  1.29it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 20%|██████████████████████▍                                                                                         | 2/10 [00:01&lt;00:06,  1.22it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 20%|██████████████████████████████████████▍                                                                                                                                                         | 2/10 [00:01&lt;00:06,  1.18it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 30%|█████████████████████████████████▌                                                                              | 3/10 [00:02&lt;00:05,  1.32it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 30%|█████████████████████████████████████████████████████████▌                                                                                                                                      | 3/10 [00:02&lt;00:05,  1.29it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 40%|████████████████████████████████████████████▊                                                                   | 4/10 [00:03&lt;00:04,  1.36it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 40%|████████████████████████████████████████████████████████████████████████████▊                                                                                                                   | 4/10 [00:03&lt;00:04,  1.32it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 50%|████████████████████████████████████████████████████████                                                        | 5/10 [00:03&lt;00:03,  1.40it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 50%|████████████████████████████████████████████████████████████████████████████████████████████████                                                                                                | 5/10 [00:03&lt;00:03,  1.54it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 60%|███████████████████████████████████████████████████████████████████▏                                            | 6/10 [00:04&lt;00:02,  1.47it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 60%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                                            | 6/10 [00:04&lt;00:02,  1.65it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 70%|██████████████████████████████████████████████████████████████████████████████▍                                 | 7/10 [00:04&lt;00:01,  1.53it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 70%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                         | 7/10 [00:04&lt;00:02,  1.44it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 80%|█████████████████████████████████████████████████████████████████████████████████████████▌                      | 8/10 [00:05&lt;00:01,  1.54it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                      | 8/10 [00:05&lt;00:01,  1.50it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 90%|████████████████████████████████████████████████████████████████████████████████████████████████████▊           | 9/10 [00:06&lt;00:00,  1.49it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 90%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                   | 9/10 [00:06&lt;00:00,  1.47it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:06&lt;00:00,  1.45it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:06&lt;00:00,  1.52it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:06&lt;00:00,  1.43it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:06&lt;00:00,  1.45it/s]
 </pre></div>
 </div>
 <div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>

doc/LectureNotes/_build/html/chapter4.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/chapter5.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/chapter6.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -767,9 +787,9 @@ predicting the target features of query instances is as follows:</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2nd degree coefficients:
-zero power:  -1.5105332296929628
-first power:  0.08398399377155011
-second power:  -0.0003701342170783489
+zero power:  -5.030164788184997
+first power:  0.001268544905013411
+second power:  -5.234332597247826e-05
 </pre></div>
 </div>
 <img alt="_images/chapter6_1_1.png" src="_images/chapter6_1_1.png" />
@@ -1632,10 +1652,10 @@ attributes at each step while growing the tree.</p>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(426, 30)
 (143, 30)
-Test set accuracy with Logistic Regression: 0.94
 </pre></div>
 </div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Test set accuracy with SVM: 0.63
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Test set accuracy with Logistic Regression: 0.94
+Test set accuracy with SVM: 0.63
 Test set accuracy with Decision Trees: 0.90
 Test set accuracy Logistic Regression with scaled data: 0.96
 Test set accuracy SVM with scaled data: 0.96

doc/LectureNotes/_build/html/chapter7.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/chapter8.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -721,10 +741,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.13035147135400782
-4.25879315330607
-[[0.86867512 2.59009792]
- [2.59009792 8.82533209]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.012423940191689783
+4.101008878523571
+[[0.89527291 2.65532045]
+ [2.65532045 8.81987609]]
 </pre></div>
 </div>
 </div>
@@ -764,10 +784,10 @@ a more brute force way. Here we scale the mean values for each column of the des
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08690184845323
-1.521422502348998
-[[1.         0.69768266]
- [0.69768266 1.        ]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08705631913312815
+1.7026908764394864
+[[1.         0.65870313]
+ [0.65870313 1.        ]]
 </pre></div>
 </div>
 </div>
@@ -796,30 +816,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 1.14550854  1.96870431]
- [ 0.79787194  3.11438414]
- [-0.18497496 -1.31315504]
- [-1.52706754 -4.97482498]
- [-1.30190897 -3.11113486]
- [-0.08421808 -1.70928399]
- [ 0.11992194 -0.07776381]
- [-0.90717653 -2.20404927]
- [ 1.05201041  5.38762019]
- [ 0.89003324  2.9195033 ]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-1.7755649  -4.56778296]
+ [-0.81015037 -2.80072356]
+ [ 0.73628249  1.95206335]
+ [ 0.97366347  1.61130099]
+ [ 0.7271324   1.97965627]
+ [ 0.36881837  0.56037913]
+ [-1.33163086 -2.59391196]
+ [-0.68953877 -1.58298728]
+ [ 0.19982428 -1.08010965]
+ [ 1.60116388  6.52211567]]
           0         1
-0  1.145509  1.968704
-1  0.797872  3.114384
-2 -0.184975 -1.313155
-3 -1.527068 -4.974825
-4 -1.301909 -3.111135
-5 -0.084218 -1.709284
-6  0.119922 -0.077764
-7 -0.907177 -2.204049
-8  1.052010  5.387620
-9  0.890033  2.919503
-          0         1
-0  1.000000  0.937057
-1  0.937057  1.000000
+0 -1.775565 -4.567783
+1 -0.810150 -2.800724
+2  0.736282  1.952063
+3  0.973663  1.611301
+4  0.727132  1.979656
+5  0.368818  0.560379
+6 -1.331631 -2.593912
+7 -0.689539 -1.582987
+8  0.199824 -1.080110
+9  1.601164  6.522116
+         0        1
+0  1.00000  0.94335
+1  0.94335  1.00000
 </pre></div>
 </div>
 </div>
@@ -876,37 +896,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>     0         1         2         3         4         5         6         7   \
 0   0.0  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   
-1   0.0  0.080345  0.078573  0.079174  0.077839  0.076679  0.070275  0.069320   
-2   0.0  0.078573  0.078146  0.079202  0.078688  0.078268  0.071580  0.071186   
-3   0.0  0.079174  0.079202  0.083342  0.083016  0.082784  0.076947  0.076648   
-4   0.0  0.077839  0.078688  0.083016  0.083260  0.083547  0.077485  0.077601   
-5   0.0  0.076679  0.078268  0.082784  0.083547  0.084312  0.078044  0.078541   
-6   0.0  0.070275  0.071580  0.076947  0.077485  0.078044  0.072947  0.073258   
-7   0.0  0.069320  0.071186  0.076648  0.077601  0.078541  0.073258  0.073882   
-8   0.0  0.068539  0.070918  0.076479  0.077813  0.079107  0.073647  0.074561   
-9   0.0  0.067922  0.070772  0.076434  0.078123  0.079747  0.074115  0.075302   
-10  0.0  0.061319  0.063395  0.068977  0.070098  0.071191  0.066686  0.067431   
-11  0.0  0.060726  0.063206  0.068843  0.070274  0.071656  0.066988  0.067974   
-12  0.0  0.060275  0.063130  0.068827  0.070547  0.072200  0.067374  0.068587   
-13  0.0  0.059956  0.063161  0.068924  0.070916  0.072824  0.067843  0.069270   
-14  0.0  0.059761  0.063294  0.069129  0.071379  0.073528  0.068394  0.070024   
+1   0.0  0.088650  0.084209  0.092689  0.091653  0.090476  0.085764  0.085378   
+2   0.0  0.084209  0.080559  0.088599  0.087876  0.087020  0.082478  0.082249   
+3   0.0  0.092689  0.088599  0.102209  0.101292  0.100209  0.097667  0.097380   
+4   0.0  0.091653  0.087876  0.101292  0.100523  0.099588  0.097017  0.096811   
+5   0.0  0.090476  0.087020  0.100209  0.099588  0.098803  0.096203  0.096078   
+6   0.0  0.085764  0.082478  0.097667  0.097017  0.096203  0.095425  0.095278   
+7   0.0  0.085378  0.082249  0.097380  0.096811  0.096078  0.095278  0.095178   
+8   0.0  0.084976  0.082002  0.097060  0.096570  0.095915  0.095090  0.095037   
+9   0.0  0.084550  0.081730  0.096700  0.096287  0.095710  0.094857  0.094849   
+10  0.0  0.077672  0.075070  0.090429  0.090002  0.089421  0.089826  0.089786   
+11  0.0  0.077490  0.074976  0.090319  0.089940  0.089408  0.089800  0.089790   
+12  0.0  0.077310  0.074882  0.090204  0.089872  0.089386  0.089763  0.089783   
+13  0.0  0.077131  0.074787  0.090081  0.089795  0.089354  0.089714  0.089763   
+14  0.0  0.076951  0.074688  0.089949  0.089708  0.089311  0.089653  0.089730   
 
           8         9         10        11        12        13        14  
 0   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  
-1   0.068539  0.067922  0.061319  0.060726  0.060275  0.059956  0.059761  
-2   0.070918  0.070772  0.063395  0.063206  0.063130  0.063161  0.063294  
-3   0.076479  0.076434  0.068977  0.068843  0.068827  0.068924  0.069129  
-4   0.077813  0.078123  0.070098  0.070274  0.070547  0.070916  0.071379  
-5   0.079107  0.079747  0.071191  0.071656  0.072200  0.072824  0.073528  
-6   0.073647  0.074115  0.066686  0.066988  0.067374  0.067843  0.068394  
-7   0.074561  0.075302  0.067431  0.067974  0.068587  0.069270  0.070024  
-8   0.075513  0.076508  0.068215  0.068985  0.069811  0.070696  0.071643  
-9   0.076508  0.077745  0.069045  0.070027  0.071055  0.072132  0.073262  
-10  0.068215  0.069045  0.061898  0.062520  0.063197  0.063933  0.064728  
-11  0.068985  0.070027  0.062520  0.063332  0.064189  0.065096  0.066054  
-12  0.069811  0.071055  0.063197  0.064189  0.065217  0.066286  0.067400  
-13  0.070696  0.072132  0.063933  0.065096  0.066286  0.067511  0.068775  
-14  0.071643  0.073262  0.064728  0.066054  0.067400  0.068775  0.070183  
+1   0.084976  0.084550  0.077672  0.077490  0.077310  0.077131  0.076951  
+2   0.082002  0.081730  0.075070  0.074976  0.074882  0.074787  0.074688  
+3   0.097060  0.096700  0.090429  0.090319  0.090204  0.090081  0.089949  
+4   0.096570  0.096287  0.090002  0.089940  0.089872  0.089795  0.089708  
+5   0.095915  0.095710  0.089421  0.089408  0.089386  0.089354  0.089311  
+6   0.095090  0.094857  0.089826  0.089800  0.089763  0.089714  0.089653  
+7   0.095037  0.094849  0.089786  0.089790  0.089783  0.089763  0.089730  
+8   0.094941  0.094797  0.089704  0.089738  0.089760  0.089769  0.089763  
+9   0.094797  0.094697  0.089576  0.089639  0.089690  0.089726  0.089748  
+10  0.089704  0.089576  0.085655  0.085690  0.085712  0.085722  0.085716  
+11  0.089738  0.089639  0.085690  0.085747  0.085790  0.085819  0.085833  
+12  0.089760  0.089690  0.085712  0.085790  0.085853  0.085902  0.085935  
+13  0.089769  0.089726  0.085722  0.085819  0.085902  0.085970  0.086021  
+14  0.089763  0.089748  0.085716  0.085833  0.085935  0.086021  0.086092  
 </pre></div>
 </div>
 </div>
@@ -1095,10 +1115,10 @@ We can write our own code or simply use either the functionaly of <strong>numpy<
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>          0         1
-0  4.032196  2.034476
-1  2.034476  1.997746
-[[4.0321956  2.03447649]
- [2.03447649 1.99774602]]
+0  3.914672  1.954823
+1  1.954823  1.963858
+[[3.91467223 1.95482298]
+ [1.95482298 1.96385798]]
 </pre></div>
 </div>
 </div>
@@ -1125,8 +1145,8 @@ Our own code here is not very elegant and asks for obvious improvements. It is t
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Centered covariance using own code
-[[4.0321956  2.03447649]
- [2.03447649 1.99774602]]
+[[3.91467223 1.95482298]
+ [1.95482298 1.96385798]]
 </pre></div>
 </div>
 <img alt="_images/chapter8_65_1.png" src="_images/chapter8_65_1.png" />
@@ -1186,16 +1206,16 @@ questions.</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Eigenvalues of Covariance matrix
-5.2895786617507
-0.7403629637766833
+5.123928000581825
+0.7546022135629743
 First eigenvector
-[0.85064942 0.52573336]
+[0.85043503 0.5260801 ]
 Second eigenvector
-[-0.52573336  0.85064942]
+[-0.5260801   0.85043503]
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Eigenvector of largest eigenvalue
-[-0.85064942 -0.52573336]
+[0.85043503 0.5260801 ]
 </pre></div>
 </div>
 </div>

doc/LectureNotes/_build/html/chapter9.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/chapteroptimization.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/clustering.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/exercisesweek34.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/exercisesweek35.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/exercisesweek36.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/exercisesweek37.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/exercisesweek38.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/exercisesweek39.html

@@ -306,6 +306,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -318,6 +333,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/exercisesweek41.html

@@ -794,15 +794,15 @@ regression.</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[4.0846409]
- [2.8352834]]
-Eigenvalues of Hessian Matrix:[0.33241952 4.39292847]
+[[3.96581896]
+ [3.03624103]]
+Eigenvalues of Hessian Matrix:[0.26120167 4.71129677]
 theta from own gd
-[[4.0846409]
- [2.8352834]]
+[[3.96581896]
+ [3.03624103]]
 theta from own sdg
-[[4.03188133]
- [2.82130663]]
+[[3.99729015]
+ [3.02646648]]
 </pre></div>
 </div>
 <img alt="_images/exercisesweek41_5_1.png" src="_images/exercisesweek41_5_1.png" />
@@ -924,14 +924,14 @@ first example shows results with ordinary leats squares.</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[3.89543945]
- [3.14433067]]
-Eigenvalues of Hessian Matrix:[0.30069427 4.55916434]
+[[4.03656288]
+ [2.95497813]]
+Eigenvalues of Hessian Matrix:[0.29560433 4.58257727]
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own gd
-[[3.89543945]
- [3.14433067]]
+[[4.03656288]
+ [2.95497813]]
 </pre></div>
 </div>
 <img alt="_images/exercisesweek41_16_2.png" src="_images/exercisesweek41_16_2.png" />
@@ -1002,73 +1002,73 @@ Eigenvalues of Hessian Matrix:[0.30069427 4.55916434]
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
 [[4.]
  [3.]]
-Eigenvalues of Hessian Matrix:[0.30043041 4.28093636]
-0 [-13.544421] [-15.11052859]
-1 [-0.26692247] [0.23040825]
-2 [-0.2481902] [0.2142385]
-3 [-0.23077255] [0.19920353]
-4 [-0.21457724] [0.18522369]
-5 [-0.19951849] [0.17222494]
-6 [-0.18551655] [0.16013842]
-7 [-0.17249725] [0.14890012]
-8 [-0.16039162] [0.13845051]
-9 [-0.14913555] [0.12873424]
-10 [-0.13866942] [0.11969984]
-11 [-0.12893778] [0.11129947]
-12 [-0.1198891] [0.10348862]
-13 [-0.11147544] [0.09622593]
-14 [-0.10365224] [0.08947292]
-15 [-0.09637807] [0.08319383]
-16 [-0.08961438] [0.0773554]
-17 [-0.08332537] [0.0719267]
-18 [-0.0774777] [0.06687898]
-19 [-0.07204042] [0.0621855]
-20 [-0.06698472] [0.0578214]
-21 [-0.06228382] [0.05376358]
-22 [-0.05791283] [0.04999052]
-23 [-0.05384858] [0.04648225]
-24 [-0.05006956] [0.04322019]
-25 [-0.04655574] [0.04018705]
-26 [-0.04328852] [0.03736678]
-27 [-0.04025059] [0.03474443]
-28 [-0.03742586] [0.03230611]
-29 [-0.03479936] [0.03003891]
+Eigenvalues of Hessian Matrix:[0.27750534 4.13217667]
+0 [-10.24816468] [-11.05638246]
+1 [-0.1602625] [0.14404536]
+2 [-0.14949972] [0.13437168]
+3 [-0.13945974] [0.12534765]
+4 [-0.13009402] [0.11692966]
+5 [-0.12135727] [0.10907699]
+6 [-0.11320726] [0.10175169]
+7 [-0.10560458] [0.09491833]
+8 [-0.09851247] [0.08854388]
+9 [-0.09189665] [0.08259753]
+10 [-0.08572513] [0.07705051]
+11 [-0.07996807] [0.07187601]
+12 [-0.07459764] [0.06704902]
+13 [-0.06958788] [0.0625462]
+14 [-0.06491455] [0.05834577]
+15 [-0.06055507] [0.05442743]
+16 [-0.05648836] [0.05077224]
+17 [-0.05269476] [0.04736252]
+18 [-0.04915593] [0.04418179]
+19 [-0.04585476] [0.04121466]
+20 [-0.04277528] [0.0384468]
+21 [-0.03990261] [0.03586482]
+22 [-0.03722287] [0.03345624]
+23 [-0.03472308] [0.03120942]
+24 [-0.03239118] [0.02911348]
+25 [-0.03021588] [0.0271583]
+26 [-0.02818667] [0.02533443]
+27 [-0.02629373] [0.02363304]
+28 [-0.02452792] [0.02204591]
+29 [-0.02288069] [0.02056537]
 theta from own gd
-[[3.89229722]
- [3.09296935]]
-0 [-0.03235719] [0.02793082]
-1 [-0.03008641] [0.02597067]
-2 [-0.02729375] [0.02356004]
-3 [-0.02454051] [0.02118344]
-4 [-0.02199232] [0.01898383]
-5 [-0.01968447] [0.01699169]
-6 [-0.01761069] [0.01520159]
-7 [-0.01575266] [0.01359774]
-8 [-0.01408975] [0.01216231]
-9 [-0.01260207] [0.01087815]
-10 [-0.01127138] [0.00972948]
-11 [-0.01008116] [0.00870208]
-12 [-0.00901661] [0.00778316]
-13 [-0.00806447] [0.00696127]
-14 [-0.00721287] [0.00622617]
-15 [-0.00645121] [0.0055687]
-16 [-0.00576997] [0.00498065]
-17 [-0.00516067] [0.0044547]
-18 [-0.00461571] [0.00398429]
-19 [-0.0041283] [0.00356356]
-20 [-0.00369236] [0.00318725]
-21 [-0.00330245] [0.00285068]
-22 [-0.00295372] [0.00254966]
-23 [-0.00264181] [0.00228042]
-24 [-0.00236284] [0.00203961]
-25 [-0.00211332] [0.00182423]
-26 [-0.00189016] [0.00163159]
-27 [-0.00169056] [0.0014593]
-28 [-0.00151204] [0.0013052]
-29 [-0.00135237] [0.00116737]
+[[3.92308585]
+ [3.06913112]]
+0 [-0.02134409] [0.01918426]
+1 [-0.01991068] [0.01789589]
+2 [-0.01814351] [0.01630755]
+3 [-0.01639489] [0.01473588]
+4 [-0.01476927] [0.01327475]
+5 [-0.01328972] [0.01194492]
+6 [-0.01195336] [0.01074379]
+7 [-0.0107497] [0.00966192]
+8 [-0.00966668] [0.0086885]
+9 [-0.00869259] [0.00781297]
+10 [-0.00781659] [0.00702562]
+11 [-0.00702885] [0.00631759]
+12 [-0.00632049] [0.00568091]
+13 [-0.00568352] [0.00510839]
+14 [-0.00511073] [0.00459357]
+15 [-0.00459568] [0.00413064]
+16 [-0.00413253] [0.00371435]
+17 [-0.00371605] [0.00334002]
+18 [-0.00334155] [0.00300342]
+19 [-0.00300479] [0.00270073]
+20 [-0.00270197] [0.00242856]
+21 [-0.00242967] [0.00218381]
+22 [-0.00218481] [0.00196372]
+23 [-0.00196462] [0.00176582]
+24 [-0.00176663] [0.00158786]
+25 [-0.00158859] [0.00142784]
+26 [-0.00142849] [0.00128394]
+27 [-0.00128453] [0.00115455]
+28 [-0.00115508] [0.00103819]
+29 [-0.00103867] [0.00093356]
 theta from own gd wth momentum
-[[3.9959739 ]
- [3.00347534]]
+[[3.99663433]
+ [3.00302509]]
 </pre></div>
 </div>
 </div>
@@ -1121,17 +1121,17 @@ theta from own gd wth momentum
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[3.96414331]
- [3.22481166]]
-Eigenvalues of Hessian Matrix:[0.26674792 4.38123077]
-0 [-15.60151468] [-17.64192473]
-1 [-1.66342634e-14] [-1.21941484e-14]
-2 [-6.41847686e-17] [1.63371136e-16]
-3 [-6.76542156e-17] [-4.78340902e-17]
-4 [-6.76542156e-17] [-4.78340902e-17]
+[[3.81924504]
+ [3.09224244]]
+Eigenvalues of Hessian Matrix:[0.31096156 4.57735536]
+0 [-15.77829399] [-18.53684024]
+1 [-3.15242624e-15] [-1.65243388e-16]
+2 [3.85975973e-17] [5.35468713e-17]
+3 [3.85975973e-17] [5.35468713e-17]
+4 [3.85975973e-17] [5.35468713e-17]
 beta from own Newton code
-[[3.96414331]
- [3.22481166]]
+[[3.81924504]
+ [3.09224244]]
 </pre></div>
 </div>
 </div>
@@ -1220,20 +1220,20 @@ beta from own Newton code
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[3.84668873]
- [3.21776599]]
-Eigenvalues of Hessian Matrix:[0.28766899 4.2084894 ]
+[[3.84019294]
+ [3.22756518]]
+Eigenvalues of Hessian Matrix:[0.2729933  4.56108455]
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own gd
-[[3.84668873]
- [3.21776599]]
+[[3.84019294]
+ [3.22756518]]
 </pre></div>
 </div>
 <img alt="_images/exercisesweek41_22_2.png" src="_images/exercisesweek41_22_2.png" />
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own sdg
-[[3.86757762]
- [3.17214797]]
+[[3.85298956]
+ [3.28358004]]
 </pre></div>
 </div>
 </div>
@@ -1315,15 +1315,15 @@ Eigenvalues of Hessian Matrix:[0.28766899 4.2084894 ]
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[3.90140097]
- [3.0552878 ]]
-Eigenvalues of Hessian Matrix:[0.32683063 4.01722503]
+[[4.1356995 ]
+ [3.09209032]]
+Eigenvalues of Hessian Matrix:[0.31362428 4.21949761]
 theta from own gd
-[[3.90137221]
- [3.055314  ]]
+[[4.13531906]
+ [3.09242193]]
 theta from own sdg with momentum
-[[3.83824223]
- [3.10556487]]
+[[4.25003261]
+ [3.05169046]]
 </pre></div>
 </div>
 </div>
@@ -1398,9 +1398,9 @@ theta from own sdg with momentum
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own AdaGrad
-[[2.00003645]
- [2.99977853]
- [4.00021916]]
+[[2.00006114]
+ [2.99959609]
+ [4.00040218]]
 </pre></div>
 </div>
 </div>
@@ -1482,9 +1482,9 @@ theta from own sdg with momentum
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own RMSprop
-[[1.99985591]
- [3.00049636]
- [3.99943837]]
+[[1.99952851]
+ [3.00380258]
+ [3.995522  ]]
 </pre></div>
 </div>
 </div>
@@ -1570,9 +1570,9 @@ theta from own sdg with momentum
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own ADAM
-[[2.00008335]
- [2.99969576]
- [4.00033679]]
+[[1.99984836]
+ [3.00097416]
+ [3.99895448]]
 </pre></div>
 </div>
 </div>
@@ -1645,7 +1645,7 @@ It provides composable transformations of Python+NumPy programs: differentiate,
   return asarray(x, dtype=self.dtype)
 </pre></div>
 </div>
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x126431130&gt;]
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x145270160&gt;]
 </pre></div>
 </div>
 <img alt="_images/exercisesweek41_39_2.png" src="_images/exercisesweek41_39_2.png" />
@@ -1680,7 +1680,7 @@ It provides composable transformations of Python+NumPy programs: differentiate,
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.collections.PathCollection at 0x117edb760&gt;
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.collections.PathCollection at 0x1452d6eb0&gt;
 </pre></div>
 </div>
 <img alt="_images/exercisesweek41_41_1.png" src="_images/exercisesweek41_41_1.png" />

doc/LectureNotes/_build/html/linalg.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -623,8 +643,8 @@ matrices and vectors.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 0.69465386  1.75956617 -0.23303727 -0.53125507  1.34598722 -1.09928714
-  1.37013105  0.79898903 -0.23663482  0.99427512]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 0.03563709  0.57852915  0.83220985  1.27108866 -0.3587467  -0.38713573
+ -0.09584387  0.5223261   1.7663967   0.94027059]
 </pre></div>
 </div>
 </div>
@@ -845,26 +865,26 @@ as (recall that we user lowercase letters for vectors and uppercase letters for
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[0.81947005 0.93619369 0.37449168 0.91933003 0.76855921 0.41820116
-  0.96853248 0.60375434 0.96015381 0.30269539]
- [0.46086448 0.16777789 0.9930742  0.10837392 0.69089532 0.94221383
-  0.53629564 0.50327198 0.33734605 0.04757138]
- [0.51069279 0.12363332 0.79171202 0.16791183 0.62617788 0.9288904
-  0.85112594 0.86519139 0.61192712 0.90842732]
- [0.82551764 0.67524588 0.02175561 0.1118933  0.42575338 0.45731379
-  0.61069681 0.40184681 0.18702469 0.71838601]
- [0.68655456 0.11747908 0.28253033 0.4591127  0.68072161 0.59372982
-  0.95343966 0.24780663 0.98740373 0.06808421]
- [0.99512017 0.14828178 0.02354386 0.90860768 0.891715   0.39039235
-  0.48151166 0.43563433 0.52657934 0.73176319]
- [0.77030637 0.00676256 0.37454707 0.5076963  0.51937727 0.46065811
-  0.65917558 0.72962885 0.99370678 0.92341148]
- [0.16292908 0.17214545 0.44995924 0.20367355 0.64885265 0.34225662
-  0.4215795  0.27933134 0.02552966 0.62908496]
- [0.8084934  0.51364117 0.4937346  0.05296475 0.69247718 0.56783103
-  0.85276538 0.52635761 0.96461948 0.67374815]
- [0.02137508 0.03177331 0.78186404 0.33096549 0.8423144  0.07745579
-  0.4619526  0.61414743 0.38460453 0.51928402]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[0.79010601 0.87637891 0.68824222 0.4636751  0.50100007 0.22715479
+  0.17865868 0.90903158 0.5736973  0.96961052]
+ [0.57293306 0.96660465 0.65525178 0.52480767 0.70159137 0.30894285
+  0.11675128 0.60321647 0.68281739 0.64952115]
+ [0.21833102 0.74761815 0.52789388 0.27530242 0.69463406 0.89961861
+  0.91864509 0.41794469 0.27403356 0.11416086]
+ [0.77596142 0.20224027 0.68830164 0.50895251 0.83741078 0.71957514
+  0.78945959 0.94466211 0.06443054 0.29474356]
+ [0.39914635 0.22706777 0.23499891 0.9794096  0.33435637 0.28614301
+  0.21641173 0.16925937 0.79086674 0.41259788]
+ [0.70408202 0.57833531 0.01817739 0.64689773 0.71380438 0.69311221
+  0.09930135 0.90168941 0.47308061 0.445128  ]
+ [0.10100211 0.60575887 0.69824402 0.06423317 0.24582593 0.97235642
+  0.21181534 0.72033728 0.77014839 0.13298019]
+ [0.25816519 0.81826799 0.19336703 0.34098895 0.10688434 0.34134773
+  0.21635399 0.57016227 0.69925648 0.01418766]
+ [0.80374623 0.58202531 0.71460518 0.66363129 0.02553865 0.7204561
+  0.34704885 0.52927353 0.02631244 0.02944974]
+ [0.57080764 0.04516434 0.15388662 0.99458998 0.2765068  0.05148401
+  0.82259916 0.05648118 0.14249052 0.96164155]]
 </pre></div>
 </div>
 </div>
@@ -924,13 +944,15 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07218473624441492
-4.346000154268618
-0.09048717285916912
-[[ 1.0974119   3.2131067   3.36880327]
- [ 3.2131067  10.51564682  9.75138064]
- [ 3.36880327  9.75138064 17.26527089]]
-[25.09487358  0.09070064  3.6927554 ]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.22370461988004753
+4.592766658048914
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.7163835161938888
+[[ 1.11935351  3.19945294  3.19001247]
+ [ 3.19945294 10.52074297  8.75737136]
+ [ 3.19001247  8.75737136 16.29078236]]
+[23.51532469  0.10801706  4.3075371 ]
 </pre></div>
 </div>
 </div>

doc/LectureNotes/_build/html/schedule.html

@@ -306,6 +306,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -318,6 +333,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/statistics.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -995,27 +1015,27 @@ uncorrelated.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>3.613893902124586
-[[18.33217312 13.09829902  1.98271562 18.53534008 -0.88860399 18.12957454
-   7.18043393  1.59698459 17.87186187 10.81979166]
- [13.09829902  9.35870701  1.41664613 13.24346138 -0.63490568 12.95354276
-   5.13040489  1.14104212 12.76940759  7.73071831]
- [ 1.98271562  1.41664613  0.21444055  2.00468914 -0.09610694  1.96080358
-   0.77659961  0.17272182  1.93293067  1.17021424]
- [18.53534008 13.24346138  2.00468914 18.74075864 -0.89845198 18.33049619
-   7.26001134  1.61468323 18.0699274  10.93970238]
- [-0.88860399 -0.63490568 -0.09610694 -0.89845198  0.04307275 -0.87878356
-  -0.3480527  -0.07740964 -0.86629161 -0.52446101]
- [18.12957454 12.95354276  1.96080358 18.33049619 -0.87878356 17.92921498
-   7.10107914  1.57933547 17.67435043 10.7002164 ]
- [ 7.18043393  5.13040489  0.77659961  7.26001134 -0.3480527   7.10107914
-   2.81246697  0.62551462  7.000137    4.23794815]
- [ 1.59698459  1.14104212  0.17272182  1.61468323 -0.07740964  1.57933547
-   0.62551462  0.13911934  1.55688514  0.94255277]
- [17.87186187 12.76940759  1.93293067 18.0699274  -0.86629161 17.67435043
-   7.000137    1.55688514 17.42310878 10.54811237]
- [10.81979166  7.73071831  1.17021424 10.93970238 -0.52446101 10.7002164
-   4.23794815  0.94255277 10.54811237  6.38592549]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>4.177786562639708
+[[ 8.02409835 11.56990537 11.96799937  6.11006373  8.01793972  4.09014451
+  12.65476419 -4.31938579 13.04431557 14.12208052]
+ [11.56990537 16.68258595 17.25659561  8.81006889 11.56102529  5.89755794
+  18.2468382  -6.22809974 18.80853029 20.36255393]
+ [11.96799937 17.25659561 17.85035562  9.11320323 11.95881375  6.10047943
+  18.8746702  -6.44239442 19.45568883 21.06318287]
+ [ 6.11006373  8.81006889  9.11320323  4.65259487  6.10537416  3.11449867
+   9.63615006 -3.2890577   9.93277949 10.75345893]
+ [ 8.01793972 11.56102529 11.95881375  6.10537416  8.01178583  4.08700526
+  12.64505146 -4.31607059 13.03430385 14.1112416 ]
+ [ 4.09014451  5.89755794  6.10047943  3.11449867  4.08700526  2.08487999
+   6.45054585 -2.20173175  6.64911288  7.19848482]
+ [12.65476419 18.2468382  18.8746702   9.63615006 12.64505146  6.45054585
+  19.95776346 -6.81208109 20.57212292 22.27186049]
+ [-4.31938579 -6.22809974 -6.44239442 -3.2890577  -4.31607059 -2.20173175
+  -6.81208109  2.32513272 -7.02177725 -7.60193996]
+ [13.04431557 18.80853029 19.45568883  9.93277949 13.03430385  6.64911288
+  20.57212292 -7.02177725 21.20539421 22.95745476]
+ [14.12208052 20.36255393 21.06318287 10.75345893 14.1112416   7.19848482
+  22.27186049 -7.60193996 22.95745476 24.85427641]]
 </pre></div>
 </div>
 </div>
@@ -1283,15 +1303,15 @@ more practically oriented methods like the blocking technique.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.0973900819831327
-4.559791959661649
-0.6333349473431832
-1.1064601390492845 11.271260217275557 17.072290639572326
-3.390334838933238 3.4951301940723654 11.033233504714921
-[[ 1.10646014  3.39033484  3.49513019]
- [ 3.39033484 11.27126022 11.0332335 ]
- [ 3.49513019 11.0332335  17.07229064]]
-[26.5020612  0.075885   2.8720648]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.061772005077293704
+4.355734685106391
+-0.03692132794542241
+1.0765856870687196 10.882972378790114 11.598549916355722
+3.274394546800107 2.662153330760193 7.985725003240627
+[[ 1.07658569  3.27439455  2.66215333]
+ [ 3.27439455 10.88297238  7.985725  ]
+ [ 2.66215333  7.985725   11.59854992]]
+[20.15422927  0.07415258  3.32972613]
 </pre></div>
 </div>
 </div>
@@ -1621,7 +1641,7 @@ assumption for approximating <span class="math notranslate nohighlight">\(\sigma
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.02214409916811925 1.073576975500551
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.0370757046153366 1.001106660107757
 </pre></div>
 </div>
 <img alt="_images/statistics_188_1.png" src="_images/statistics_188_1.png" />

doc/LectureNotes/_build/html/teachers.html

@@ -306,6 +306,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -318,6 +333,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/textbooks.html

@@ -306,6 +306,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -318,6 +333,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/week34.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -1683,8 +1703,8 @@ developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly he
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[-0.60588173  1.59884169 -1.56922102 -1.32068736 -0.21631786  0.37600869
- -0.14841322 -0.83655756  0.88809826 -0.85625134]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.51262599  0.63980912 -1.25680702  0.97680846 -1.33095972 -0.41396339
+ -0.81478187 -0.6087346   2.11164003 -1.21061589]
 </pre></div>
 </div>
 </div>
@@ -1909,26 +1929,26 @@ lowercase letters for vectors and uppercase letters for matrices)</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[0.32641434 0.35731585 0.64796751 0.41146753 0.67561021 0.53312817
-  0.3910042  0.88157423 0.4576811  0.48004784]
- [0.70469324 0.28496213 0.84862355 0.6988905  0.95890587 0.19849513
-  0.89058005 0.51546825 0.88289263 0.06926192]
- [0.38198866 0.33000573 0.80740939 0.54935794 0.38934047 0.87740526
-  0.45053025 0.27231287 0.7070883  0.7989356 ]
- [0.4198932  0.3727791  0.95400323 0.86459987 0.2666905  0.13564988
-  0.97498674 0.9450635  0.6383903  0.57803254]
- [0.13896095 0.13663125 0.68826552 0.13729154 0.91672129 0.08266769
-  0.88639567 0.16407038 0.36353321 0.81007381]
- [0.31849289 0.68735473 0.1767857  0.42873361 0.44454123 0.21333766
-  0.94285762 0.72710494 0.37153115 0.21070843]
- [0.11930916 0.28021598 0.69566966 0.98770503 0.88653291 0.82161167
-  0.90114639 0.7127128  0.97486336 0.26152075]
- [0.55386681 0.37919989 0.57468142 0.35980374 0.7150195  0.70499955
-  0.9647801  0.63142399 0.97512176 0.97570392]
- [0.24613581 0.62573269 0.41487642 0.42095725 0.51447004 0.41869784
-  0.34483955 0.55582742 0.85711016 0.17739525]
- [0.89533642 0.03382942 0.918785   0.79718864 0.64361375 0.4843772
-  0.33532886 0.13164176 0.63209435 0.39279291]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[0.1169204  0.51615779 0.40961688 0.169299   0.08009874 0.67925887
+  0.8475889  0.92080432 0.07712724 0.2863391 ]
+ [0.36161658 0.84155431 0.70135856 0.1576057  0.04686491 0.67511113
+  0.29593305 0.22946401 0.78385675 0.20527785]
+ [0.66575697 0.37717637 0.52407775 0.55094784 0.68989446 0.30013135
+  0.39991048 0.20300793 0.25294371 0.91433102]
+ [0.05080769 0.92665802 0.77039278 0.13455019 0.89692576 0.09621323
+  0.48511333 0.8529175  0.32738537 0.12206812]
+ [0.98108401 0.73397147 0.62288579 0.66003032 0.18712313 0.63307537
+  0.2032806  0.17418673 0.06061276 0.92991181]
+ [0.53480404 0.69484973 0.09821823 0.93019783 0.34478594 0.18646225
+  0.11861803 0.25646067 0.55225408 0.84907109]
+ [0.50352245 0.92678221 0.27037635 0.9833205  0.84985833 0.82844656
+  0.34112554 0.9306628  0.89155606 0.24149532]
+ [0.37137157 0.65751456 0.63693246 0.25068519 0.75674251 0.43724406
+  0.34131583 0.74180248 0.63801791 0.76426396]
+ [0.2311959  0.77594586 0.52606333 0.54222783 0.86434639 0.72364915
+  0.4008393  0.68827947 0.56408898 0.68640031]
+ [0.90137794 0.02599188 0.40848657 0.94114646 0.67199457 0.02124568
+  0.32717946 0.59030403 0.59188296 0.81707832]]
 </pre></div>
 </div>
 </div>
@@ -1983,13 +2003,13 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.050315327923114654
-3.7539148284817965
--0.09246293455466892
-[[ 0.90894281  2.8691787   2.58589719]
- [ 2.8691787  10.23481471  8.16535252]
- [ 2.58589719  8.16535252 12.44536331]]
-[20.33742926  0.08458591  3.16710567]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.04372067685794603
+3.777112373675558
+-0.028188673105960467
+[[ 1.0680225   3.36841864  2.42592679]
+ [ 3.36841864 11.49312535  7.50233673]
+ [ 2.42592679  7.50233673  7.94272259]]
+[18.418656    0.05992237  2.02529207]
 </pre></div>
 </div>
 </div>
@@ -2214,7 +2234,7 @@ Name: Aragorn, dtype: object
 <div class="cell_output docutils container">
 <div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
 <span class="ne">AttributeError</span><span class="g g-Whitespace">                            </span>Traceback (most recent call last)
-<span class="nn">/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95519/1326197715.py</span> in <span class="ni">?</span><span class="nt">()</span>
+<span class="nn">/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22556/1326197715.py</span> in <span class="ni">?</span><span class="nt">()</span>
 <span class="ne">----&gt; </span><span class="mi">6</span> <span class="n">new_hobbit</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;First Name&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;Peregrin&quot;</span><span class="p">],</span>
 <span class="g g-Whitespace">      </span><span class="mi">7</span>               <span class="s1">&#39;Last Name&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;Took&quot;</span><span class="p">],</span>
 <span class="g g-Whitespace">      </span><span class="mi">8</span>               <span class="s1">&#39;Place of birth&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;Shire&quot;</span><span class="p">],</span>

doc/LectureNotes/_build/html/week35.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -1651,7 +1671,7 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9952505213910134
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9960887274532307
 </pre></div>
 </div>
 </div>
@@ -1668,7 +1688,7 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.008753288788081405
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.010148621093080332
 </pre></div>
 </div>
 </div>
@@ -1683,31 +1703,23 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[8.90304177e-02 3.21655059e-02 1.15924557e-02 1.83823317e-02
- 8.19559737e-03 1.66018725e-02 1.79135536e-03 1.03313259e-01
- 7.69182892e-04 1.29306584e-02 1.21565007e-02 2.83463634e-03
- 3.03538673e-02 2.37415625e-02 1.56790195e-02 9.03400724e-03
- 1.35613536e-02 5.35506347e-02 1.01123792e-02 4.10604579e-02
- 2.63898112e-02 1.94766419e-02 3.81425129e-02 3.27482829e-02
- 5.12829994e-03 6.02901273e-03 8.26321660e-02 4.04504728e-02
- 2.20601797e-02 4.62113349e-03 9.03476611e-04 4.87494456e-02
- 3.82060913e-03 2.53729411e-02 2.38612299e-02 1.59355752e-02
- 3.60160003e-03 1.65738717e-02 2.98947674e-02 5.18501900e-03
- 9.36303682e-03 4.81218742e-02 1.49392067e-02 4.88551766e-03
- 2.17643975e-02 2.20608548e-04 1.90135464e-02 2.74291603e-02
- 1.23344210e-02 6.03309191e-03 1.57252451e-02 9.02612988e-03
- 3.32084559e-02 3.76692036e-03 2.87169607e-02 4.85551266e-02
- 1.48826894e-02 6.41842093e-04 1.89017198e-02 3.49584063e-02
- 1.77652198e-02 6.38298234e-03 1.05034088e-03 1.99753321e-02
- 5.52031552e-03 8.22217237e-03 6.86192682e-02 8.40354798e-03
- 1.29491144e-02 7.44658658e-03 1.00731392e-02 9.52284329e-02
- 1.51437058e-02 2.00002585e-05 2.37700967e-02 1.95166920e-02
- 4.82376174e-02 3.73986200e-02 4.84707251e-02 8.76887316e-02
- 2.74724414e-02 5.14825560e-03 1.26254957e-02 2.81042619e-02
- 2.11265643e-02 2.52301447e-03 3.13819592e-02 2.93900569e-02
- 3.65720152e-02 1.02850506e-02 4.85945208e-02 2.79870689e-02
- 3.12846660e-02 6.17869861e-02 9.09590269e-03 1.11715109e-02
- 3.62863106e-02 1.21277816e-02 9.05665429e-03 4.85293303e-02]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.01006224 0.02667406 0.0272743  0.00100202 0.01480927 0.00410183
+ 0.02088307 0.02506901 0.00463124 0.00672505 0.0304378  0.00076521
+ 0.05364579 0.01232366 0.00969623 0.00677215 0.02239876 0.03858916
+ 0.00923474 0.00848739 0.01701429 0.07172932 0.08858739 0.04575775
+ 0.02988432 0.00988356 0.01546126 0.00033415 0.03201922 0.01425384
+ 0.00575681 0.01882218 0.05853984 0.00528252 0.03254614 0.03092729
+ 0.01094916 0.05696574 0.03412067 0.02444266 0.05749111 0.0689303
+ 0.00736737 0.02104639 0.00274212 0.00588836 0.05043958 0.01456865
+ 0.01813363 0.06019833 0.01078008 0.01059242 0.02369044 0.02692965
+ 0.00657601 0.01218403 0.03745816 0.05363722 0.00561212 0.03820746
+ 0.00988992 0.00774376 0.03426412 0.01323341 0.02387182 0.01151556
+ 0.01097287 0.07292363 0.02846506 0.04186033 0.00836649 0.00340452
+ 0.06200455 0.03246707 0.02987496 0.00355323 0.01740381 0.01196506
+ 0.02635861 0.07487128 0.08472879 0.0073544  0.01150437 0.00571884
+ 0.02025574 0.0014028  0.01512884 0.02146636 0.05097344 0.0284405
+ 0.06151386 0.00737863 0.04452918 0.03906948 0.01163942 0.07468007
+ 0.01647074 0.0096667  0.00369201 0.0168171 ]
 </pre></div>
 </div>
 </div>
@@ -1776,15 +1788,15 @@ but now splitting the data into a training set and a test set.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.05054319 -0.48521055  6.95338273 -2.63619709  1.0524497 ]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.97802245  0.57331229  2.49761526  3.47609206 -1.5187643 ]
 Training R2
-0.9959308805732706
+0.995702810640425
 Training MSE
-0.009211602191395454
+0.007370297974992432
 Test R2
-0.9955318336036834
+0.9950019477819025
 Test MSE
-0.011818646101922625
+0.009880124918446542
 </pre></div>
 </div>
 </div>
@@ -2448,9 +2460,7 @@ the aims is to reproduce Figure 2.11 of <a class="reference external" href="http
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.00
 R2 score before scaling 1.00
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Feature min values before scaling:
+Feature min values before scaling:
  [1.00000000e+00 6.97906022e-03 2.43639284e-03 4.87072815e-05
  1.70037324e-05 5.93601008e-06 3.39931051e-07 1.18670072e-07
  4.14277718e-08 1.44624525e-08 2.37239927e-09 8.28205578e-10

doc/LectureNotes/_build/html/week36.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/week37.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -1639,7 +1659,7 @@ theorem.</p>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Bootstrap Statistics :
 original           bias      std. error
- 99.8485  15.0562        99.8488        0.149353
+ 100.257   14.853        100.257        0.149111
 </pre></div>
 </div>
 </div>
@@ -1859,9 +1879,7 @@ Error: 0.08426840630693411
 Bias^2: 0.0796891867672603
 Var: 0.004579219539673834
 0.08426840630693411 &gt;= 0.0796891867672603 + 0.004579219539673834 = 0.08426840630693413
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 2
+Polynomial degree: 2
 Error: 0.10398646080125035
 Bias^2: 0.1007711427354898
 Var: 0.0032153180657605116
@@ -1871,7 +1889,9 @@ Error: 0.06547790180152355
 Bias^2: 0.06208238634231949
 Var: 0.0033955154592040936
 0.06547790180152355 &gt;= 0.06208238634231949 + 0.0033955154592040936 = 0.06547790180152359
-Polynomial degree: 4
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 4
 Error: 0.06844519414009445
 Bias^2: 0.06453579006728324
 Var: 0.003909404072811226
@@ -1881,9 +1901,7 @@ Error: 0.05227921801205686
 Bias^2: 0.0481872773043029
 Var: 0.004091940707753939
 0.05227921801205686 &gt;= 0.0481872773043029 + 0.004091940707753939 = 0.052279218012056844
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 6
+Polynomial degree: 6
 Error: 0.037813671417389005
 Bias^2: 0.033657685071527665
 Var: 0.00415598634586135
@@ -1893,19 +1911,21 @@ Error: 0.02760977349102253
 Bias^2: 0.022999498260366312
 Var: 0.004610275230656212
 0.02760977349102253 &gt;= 0.022999498260366312 + 0.004610275230656212 = 0.027609773491022525
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 8
+Polynomial degree: 8
 Error: 0.017355848195593347
 Bias^2: 0.010331721306655127
 Var: 0.007024126888938232
 0.017355848195593347 &gt;= 0.010331721306655127 + 0.007024126888938232 = 0.01735584819559336
-Polynomial degree: 9
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 9
 Error: 0.02660572763718093
 Bias^2: 0.010018312644137363
 Var: 0.016587414993043573
 0.02660572763718093 &gt;= 0.010018312644137363 + 0.016587414993043573 = 0.026605727637180936
-Polynomial degree: 10
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 10
 Error: 0.021592704588025025
 Bias^2: 0.010516485576645508
 Var: 0.011076219011379514
@@ -1915,9 +1935,7 @@ Error: 0.07160048164233104
 Bias^2: 0.014436800088904942
 Var: 0.05716368155342608
 0.07160048164233104 &gt;= 0.014436800088904942 + 0.05716368155342608 = 0.07160048164233102
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 12
+Polynomial degree: 12
 Error: 0.11547777218872497
 Bias^2: 0.01628578269596628
 Var: 0.09919198949275869
@@ -1929,7 +1947,7 @@ Var: 0.20867052175034223
 0.22842468702219465 &gt;= 0.01975416527185249 + 0.20867052175034223 = 0.2284246870221947
 </pre></div>
 </div>
-<img alt="_images/week37_139_5.png" src="_images/week37_139_5.png" />
+<img alt="_images/week37_139_4.png" src="_images/week37_139_4.png" />
 </div>
 </div>
 </div>
@@ -2336,33 +2354,33 @@ Mean squared error on test data: 1184.60929685
 Degree of polynomial:  23
 Mean squared error on training data: 0.00089193
 Mean squared error on test data: 3892.17483760
-Degree of polynomial:  24
-Mean squared error on training data: 0.00083355
-Mean squared error on test data: 1332.46736215
 </pre></div>
 </div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  25
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  24
+Mean squared error on training data: 0.00083355
+Mean squared error on test data: 1332.46736215
+Degree of polynomial:  25
 Mean squared error on training data: 0.00079904
 Mean squared error on test data: 7577.76690383
 Degree of polynomial:  26
 Mean squared error on training data: 0.00075590
 Mean squared error on test data: 1079.36895644
-Degree of polynomial:  27
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  27
 Mean squared error on training data: 0.00068091
 Mean squared error on test data: 3207.25343155
 Degree of polynomial:  28
 Mean squared error on training data: 0.00063362
 Mean squared error on test data: 674.79633065
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  29
+Degree of polynomial:  29
 Mean squared error on training data: 0.00063866
 Mean squared error on test data: 3099.60342978
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95542/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22575/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
   plt.plot(polynomial, np.log10(trainingerror), label=&#39;Training Error&#39;)
-/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95542/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
+/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22575/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
   plt.plot(polynomial, np.log10(testerror), label=&#39;Test Error&#39;)
 </pre></div>
 </div>
@@ -2447,7 +2465,7 @@ Mean squared error on test data: 3099.60342978
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_95542/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_22575/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
   plt.plot(polynomial, np.log10(estimated_mse_sklearn), label=&#39;Test Error&#39;)
 </pre></div>
 </div>

doc/LectureNotes/_build/html/week38.html

@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>

doc/LectureNotes/_build/html/week39.html

@@ -55,7 +55,7 @@ const thebe_selector_output = ".output, .cell_output"
     <script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
     <link rel="index" title="Index" href="genindex.html" />
     <link rel="search" title="Search" href="search.html" />
-    <link rel="next" title="Project 1 on Machine Learning, deadline October 7 (midnight), 2024" href="project1.html" />
+    <link rel="next" title="Week 40: Gradient descent methods (continued) and start Neural networks" href="week40.html" />
     <link rel="prev" title="Exercises week 39" href="exercisesweek39.html" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
     <meta name="docsearch:language" content="None">
@@ -308,6 +308,21 @@ const thebe_selector_output = ".output, .cell_output"
    Week 39: Optimization and  Gradient Methods
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week40.html">
+   Week 40: Gradient descent methods (continued) and start Neural networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
 </ul>
 <p aria-level="2" class="caption" role="heading">
  <span class="caption-text">
@@ -320,6 +335,11 @@ const thebe_selector_output = ".output, .cell_output"
    Project 1 on Machine Learning, deadline October 7 (midnight), 2024
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -1782,7 +1802,7 @@ which equals</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x11f13fb20&gt;
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x12aaa0b20&gt;
 </pre></div>
 </div>
 <img alt="_images/week39_82_1.png" src="_images/week39_82_1.png" />
@@ -1840,7 +1860,7 @@ which equals</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x12e13a850&gt;]
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x12b14fa90&gt;]
 </pre></div>
 </div>
 <img alt="_images/week39_90_1.png" src="_images/week39_90_1.png" />
@@ -2134,11 +2154,11 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Eigenvalues of Hessian Matrix:[0.24602146 5.15830902]
-[[3.94388948]
- [3.14880915]]
-[[3.94388948]
- [3.14880915]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Eigenvalues of Hessian Matrix:[0.29950088 4.27458376]
+[[3.66841959]
+ [3.26280614]]
+[[3.66841959]
+ [3.26280614]]
 </pre></div>
 </div>
 <img alt="_images/week39_153_1.png" src="_images/week39_153_1.png" />
@@ -2169,9 +2189,9 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.1567286 ]
- [2.83641435]]
-[4.12466453] [2.80907609]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.06267057]
+ [2.86868711]]
+[4.05285677] [2.86799623]
 </pre></div>
 </div>
 </div>
@@ -2271,11 +2291,11 @@ minimum of this function.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Eigenvalues of Hessian Matrix:[0.29902518 4.33006628]
-[[3.73252708]
- [3.18093549]]
-[[3.73265885]
- [3.1808228 ]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Eigenvalues of Hessian Matrix:[0.24719968 4.31175179]
+[[4.07235641]
+ [3.05476114]]
+[[4.06908518]
+ [3.05761244]]
 </pre></div>
 </div>
 <img alt="_images/week39_166_1.png" src="_images/week39_166_1.png" />
@@ -3851,13 +3871,11 @@ Eigenvalues of Hessian Matrix:[0.29860173 3.8931686 ]
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own gd
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.0586484]
+[[4.0586484]
  [3.0718316]]
 </pre></div>
 </div>
-<img alt="_images/week39_269_3.png" src="_images/week39_269_3.png" />
+<img alt="_images/week39_269_2.png" src="_images/week39_269_2.png" />
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own sdg
 [[4.02496085]
  [3.12081773]]
@@ -4299,10 +4317,10 @@ It provides composable transformations of Python+NumPy programs: differentiate,
             <p class="prev-next-title">Exercises week 39</p>
         </div>
     </a>
-    <a class='right-next' id="next-link" href="project1.html" title="next page">
+    <a class='right-next' id="next-link" href="week40.html" title="next page">
     <div class="prev-next-info">
         <p class="prev-next-subtitle">next</p>
-        <p class="prev-next-title">Project 1 on Machine Learning, deadline October 7 (midnight), 2024</p>
+        <p class="prev-next-title">Week 40: Gradient descent methods (continued) and start Neural networks</p>
     </div>
     <i class="fas fa-angle-right"></i>
     </a>

doc/LectureNotes/_build/html/week40.html

@@ -55,6 +55,7 @@ const thebe_selector_output = ".output, .cell_output"
     <script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
     <link rel="index" title="Index" href="genindex.html" />
     <link rel="search" title="Search" href="search.html" />
+    <link rel="next" title="Exercises week 41" href="exercisesweek41.html" />
     <link rel="prev" title="Week 39: Optimization and Gradient Methods" href="week39.html" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
     <meta name="docsearch:language" content="None">
@@ -312,6 +313,33 @@ const thebe_selector_output = ".output, .cell_output"
    Week 40: Gradient descent methods (continued) and start Neural networks
   </a>
  </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="exercisesweek41.html">
+   Exercises week 41
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="week41.html">
+   Week 41 Neural networks and constructing a neural network code
+  </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+  Projects
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+  <a class="reference internal" href="project1.html">
+   Project 1 on Machine Learning, deadline October 7 (midnight), 2024
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
 </ul>
 
     </div>
@@ -1276,17 +1304,17 @@ We summarize some of these here for the methods we hvae studied in project one,
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Parameters for OLS using gradient descent
-[[4.14721582]
- [2.54760497]
- [5.2212296 ]]
+[[4.26604611]
+ [2.29234681]
+ [5.33329216]]
 Parameters for Ridge using gradient descent
-[[3.75424998]
- [3.49608088]
- [4.78010668]]
+[[3.6161104 ]
+ [3.78762558]
+ [4.65410649]]
 Parameters for Lasso using gradient descent
-[[3.62716271]
- [3.82146046]
- [4.64516328]]
+[[4.24335376]
+ [2.36219301]
+ [5.29669411]]
 </pre></div>
 </div>
 </div>
@@ -1338,11 +1366,11 @@ Parameters for Lasso using gradient descent
 [[4.]
  [3.]
  [5.]]
-0 [-25.77106886] [-35.02189606]
-1 [3.49285045e-13] [4.71505132e-13]
-2 [1.24344979e-16] [3.86639832e-16]
-3 [9.05941988e-16] [1.48786826e-15]
-4 [-7.99360578e-16] [-1.35823372e-15]
+0 [-26.28886314] [-34.64721597]
+1 [-1.83231208e-13] [-1.80848093e-13]
+2 [6.92779167e-16] [1.22835717e-15]
+3 [-1.3500312e-15] [-1.8110093e-15]
+4 [7.46069873e-16] [9.30442002e-16]
 beta from own Newton code
 [[4.]
  [3.]
@@ -1603,9 +1631,6 @@ function.</p>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>gamma_j after 500 epochs: 9.97108e-05
 </pre></div>
 </div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
-</pre></div>
-</div>
 </div>
 </div>
 </div>
@@ -1687,15 +1712,15 @@ function.</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[4.16180248]
- [2.8250103 ]]
-Eigenvalues of Hessian Matrix:[0.26646852 4.68110474]
+[[4.36014743]
+ [2.76030841]]
+Eigenvalues of Hessian Matrix:[0.36278226 3.73204357]
 theta from own gd
-[[4.16180248]
- [2.8250103 ]]
+[[4.36014743]
+ [2.76030841]]
 theta from own sdg
-[[4.16257872]
- [2.80576215]]
+[[4.34582863]
+ [2.81684805]]
 </pre></div>
 </div>
 <img alt="_images/week40_34_1.png" src="_images/week40_34_1.png" />
@@ -2409,12 +2434,12 @@ first example shows results with ordinary leats squares.</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[3.98246764]
- [3.04018043]]
-Eigenvalues of Hessian Matrix:[0.31541884 4.4735968 ]
+[[3.85068028]
+ [3.09808149]]
+Eigenvalues of Hessian Matrix:[0.31897935 3.95770298]
 theta from own gd
-[[3.98246764]
- [3.04018043]]
+[[3.85068028]
+ [3.09808149]]
 </pre></div>
 </div>
 <img alt="_images/week40_100_1.png" src="_images/week40_100_1.png" />
@@ -2483,75 +2508,77 @@ theta from own gd
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[4.]
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.]
  [3.]]
-Eigenvalues of Hessian Matrix:[0.32799141 4.41229845]
-0 [-16.27698328] [-18.5653087]
-1 [-0.44867801] [0.37354121]
-2 [-0.41532521] [0.34577375]
-3 [-0.38445171] [0.32007041]
-4 [-0.35587321] [0.29627774]
-5 [-0.32941912] [0.27425372]
-6 [-0.30493151] [0.25386687]
-7 [-0.2822642] [0.23499549]
-8 [-0.26128189] [0.21752693]
-9 [-0.24185931] [0.20135691]
-10 [-0.22388052] [0.1863889]
-11 [-0.2072382] [0.17253354]
-12 [-0.191833] [0.15970814]
-13 [-0.17757295] [0.14783612]
-14 [-0.16437294] [0.13684661]
-15 [-0.15215415] [0.12667402]
-16 [-0.14084366] [0.11725761]
-17 [-0.13037395] [0.10854118]
-18 [-0.12068251] [0.1004727]
-19 [-0.11171148] [0.09300398]
-20 [-0.10340733] [0.08609047]
-21 [-0.09572047] [0.07969087]
-22 [-0.08860502] [0.07376699]
-23 [-0.0820185] [0.06828347]
-24 [-0.0759216] [0.06320757]
-25 [-0.07027791] [0.05850899]
-26 [-0.06505375] [0.05415968]
-27 [-0.06021793] [0.05013368]
-28 [-0.05574159] [0.04640695]
-29 [-0.051598] [0.04295726]
+Eigenvalues of Hessian Matrix:[0.35713539 3.88632765]
+0 [-9.01615836] [-9.6932681]
+1 [0.01454469] [-0.01357366]
+2 [0.0132081] [-0.0123263]
+3 [0.01199434] [-0.01119357]
+4 [0.01089212] [-0.01016493]
+5 [0.00989118] [-0.00923082]
+6 [0.00898223] [-0.00838255]
+7 [0.0081568] [-0.00761224]
+8 [0.00740723] [-0.00691271]
+9 [0.00672654] [-0.00627746]
+10 [0.0061084] [-0.00570059]
+11 [0.00554707] [-0.00517673]
+12 [0.00503732] [-0.00470102]
+13 [0.00457441] [-0.00426902]
+14 [0.00415405] [-0.00387671]
+15 [0.00377231] [-0.00352046]
+16 [0.00342565] [-0.00319695]
+17 [0.00311085] [-0.00290316]
+18 [0.00282498] [-0.00263638]
+19 [0.00256537] [-0.0023941]
+20 [0.00232963] [-0.0021741]
+21 [0.00211555] [-0.00197431]
+22 [0.00192114] [-0.00179288]
+23 [0.00174459] [-0.00162812]
+24 [0.00158427] [-0.0014785]
+25 [0.00143869] [-0.00134264]
+26 [0.00130648] [-0.00121925]
+27 [0.00118642] [-0.00110721]
+28 [0.00107739] [-0.00100546]
+29 [0.00097839] [-0.00091307]
 theta from own gd
-[[3.85437905]
- [3.12123488]]
-0 [-0.04776242] [0.039764]
-1 [-0.04421197] [0.03680811]
-2 [-0.0398603] [0.03318519]
-3 [-0.03559176] [0.02963147]
-4 [-0.03166546] [0.02636268]
-5 [-0.02813369] [0.02342235]
-6 [-0.02498282] [0.02079913]
-7 [-0.02218045] [0.01846605]
-8 [-0.01969093] [0.01639344]
-9 [-0.01748034] [0.01455304]
-10 [-0.01551775] [0.0129191]
-11 [-0.01377545] [0.01146857]
-12 [-0.01222875] [0.01018089]
-13 [-0.01085571] [0.00903778]
-14 [-0.00963683] [0.00802302]
-15 [-0.0085548] [0.00712219]
-16 [-0.00759427] [0.00632251]
-17 [-0.00674158] [0.00561262]
-18 [-0.00598464] [0.00498243]
-19 [-0.00531268] [0.004423]
-20 [-0.00471617] [0.00392639]
-21 [-0.00418664] [0.00348553]
-22 [-0.00371656] [0.00309418]
-23 [-0.00329927] [0.00274676]
-24 [-0.00292882] [0.00243835]
-25 [-0.00259997] [0.00216458]
-26 [-0.00230805] [0.00192154]
-27 [-0.0020489] [0.00170579]
-28 [-0.00181885] [0.00151426]
-29 [-0.00161463] [0.00134424]
+[[4.00248779]
+ [2.9976783 ]]
+0 [0.00088848] [-0.00082916]
+1 [0.00080683] [-0.00075296]
+2 [0.00070819] [-0.00066091]
+3 [0.00061352] [-0.00057256]
+4 [0.00052874] [-0.00049344]
+5 [0.00045472] [-0.00042436]
+6 [0.00039072] [-0.00036464]
+7 [0.00033562] [-0.00031321]
+8 [0.00028825] [-0.000269]
+9 [0.00024755] [-0.00023102]
+10 [0.00021259] [-0.0001984]
+11 [0.00018256] [-0.00017038]
+12 [0.00015678] [-0.00014631]
+13 [0.00013464] [-0.00012565]
+14 [0.00011562] [-0.0001079]
+15 [9.92929225e-05] [-9.26639089e-05]
+16 [8.52694246e-05] [-7.95766504e-05]
+17 [7.32265127e-05] [-6.83377498e-05]
+18 [6.28844641e-05] [-5.86861591e-05]
+19 [5.40030606e-05] [-5.03976976e-05]
+20 [4.63760101e-05] [-4.32798457e-05]
+21 [3.98261559e-05] [-3.71672742e-05]
+22 [3.42013616e-05] [-3.19180036e-05]
+23 [2.93709777e-05] [-2.74101067e-05]
+24 [2.52228066e-05] [-2.35388766e-05]
+25 [2.1660497e-05] [-2.02143946e-05]
+26 [1.86013055e-05] [-1.73594414e-05]
+27 [1.59741748e-05] [-1.49077038e-05]
+28 [1.37180834e-05] [-1.2802234e-05]
+29 [1.17806281e-05] [-1.09941275e-05]
 theta from own gd wth momentum
-[[3.99562995]
- [3.00363823]]
+[[4.00002833]
+ [2.99997356]]
 </pre></div>
 </div>
 </div>
@@ -2640,18 +2667,20 @@ theta from own gd wth momentum
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[4.10426556]
- [2.93942872]]
-Eigenvalues of Hessian Matrix:[0.31367041 4.07517385]
-theta from own gd
-[[4.10426556]
- [2.93942872]]
+[[3.91650453]
+ [2.94495682]]
+Eigenvalues of Hessian Matrix:[0.34862407 4.10453899]
 </pre></div>
 </div>
-<img alt="_images/week40_104_1.png" src="_images/week40_104_1.png" />
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own gd
+[[3.91650453]
+ [2.94495682]]
+</pre></div>
+</div>
+<img alt="_images/week40_104_2.png" src="_images/week40_104_2.png" />
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own sdg
-[[4.08587209]
- [2.96301764]]
+[[3.8901199 ]
+ [2.92458892]]
 </pre></div>
 </div>
 </div>
@@ -2733,17 +2762,15 @@ theta from own gd
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[4.33528448]
- [2.81889188]]
-Eigenvalues of Hessian Matrix:[0.30916402 4.51611732]
+[[4.05785974]
+ [2.95842106]]
+Eigenvalues of Hessian Matrix:[0.29678339 4.37215356]
 theta from own gd
-[[4.33477019]
- [2.81931348]]
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own sdg with momentum
-[[4.40800396]
- [2.78459458]]
+[[4.05738629]
+ [2.95882223]]
+theta from own sdg with momentum
+[[4.07489511]
+ [2.90281987]]
 </pre></div>
 </div>
 </div>
@@ -2812,9 +2839,9 @@ theta from own gd
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own AdaGrad
-[[2.00003828]
- [2.99979896]
- [4.00019258]]
+[[1.99994537]
+ [3.00034209]
+ [3.99966798]]
 </pre></div>
 </div>
 </div>
@@ -2890,9 +2917,9 @@ theta from own gd
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own RMSprop
-[[2.01129731]
- [3.01249445]
- [4.00885858]]
+[[1.99975636]
+ [3.00348281]
+ [3.99607299]]
 </pre></div>
 </div>
 </div>
@@ -2972,9 +2999,9 @@ theta from own gd
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own ADAM
-[[2.00001716]
- [2.99992107]
- [4.00007706]]
+[[2.00002276]
+ [2.99985884]
+ [4.00009004]]
 </pre></div>
 </div>
 </div>
@@ -3095,7 +3122,7 @@ It provides composable transformations of Python+NumPy programs: differentiate,
   return asarray(x, dtype=self.dtype)
 </pre></div>
 </div>
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x11753f700&gt;]
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x11bcee130&gt;]
 </pre></div>
 </div>
 <img alt="_images/week40_120_2.png" src="_images/week40_120_2.png" />
@@ -3130,7 +3157,7 @@ It provides composable transformations of Python+NumPy programs: differentiate,
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.collections.PathCollection at 0x11bc3a1f0&gt;
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;matplotlib.collections.PathCollection at 0x11bdd4640&gt;
 </pre></div>
 </div>
 <img alt="_images/week40_122_1.png" src="_images/week40_122_1.png" />
@@ -3810,6 +3837,13 @@ become the most popular for <em>deep neural networks</em></p>
             <p class="prev-next-title">Week 39: Optimization and  Gradient Methods</p>
         </div>
     </a>
+    <a class='right-next' id="next-link" href="exercisesweek41.html" title="next page">
+    <div class="prev-next-info">
+        <p class="prev-next-subtitle">next</p>
+        <p class="prev-next-title">Exercises week 41</p>
+    </div>
+    <i class="fas fa-angle-right"></i>
+    </a>
 </div>
             
         </div>

doc/LectureNotes/_build/html/week41.html

@@ -55,6 +55,7 @@ const thebe_selector_output = ".output, .cell_output"
     <script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
     <link rel="index" title="Index" href="genindex.html" />
     <link rel="search" title="Search" href="search.html" />
+    <link rel="next" title="Project 1 on Machine Learning, deadline October 7 (midnight), 2024" href="project1.html" />
     <link rel="prev" title="Exercises week 41" href="exercisesweek41.html" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
     <meta name="docsearch:language" content="None">
@@ -323,6 +324,23 @@ const thebe_selector_output = ".output, .cell_output"
   </a>
  </li>
 </ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+  Projects
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+  <a class="reference internal" href="project1.html">
+   Project 1 on Machine Learning, deadline October 7 (midnight), 2024
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project2.html">
+   Project 2 on Machine Learning, deadline November 4 (Midnight)
+  </a>
+ </li>
+</ul>
 
     </div>
 </nav> <!-- To handle the deprecated key -->
@@ -3592,6 +3610,13 @@ features).</p>
             <p class="prev-next-title">Exercises week 41</p>
         </div>
     </a>
+    <a class='right-next' id="next-link" href="project1.html" title="next page">
+    <div class="prev-next-info">
+        <p class="prev-next-subtitle">next</p>
+        <p class="prev-next-title">Project 1 on Machine Learning, deadline October 7 (midnight), 2024</p>
+    </div>
+    <i class="fas fa-angle-right"></i>
+    </a>
 </div>
             
         </div>