diff --git a/doc/pub/week42/ipynb/figures/adagrad.png b/doc/pub/week42/ipynb/figures/adagrad.png deleted file mode 100644 index 97a9cf908..000000000 Binary files a/doc/pub/week42/ipynb/figures/adagrad.png and /dev/null differ diff --git a/doc/pub/week42/ipynb/figures/adam.png b/doc/pub/week42/ipynb/figures/adam.png deleted file mode 100644 index a3a39f025..000000000 Binary files a/doc/pub/week42/ipynb/figures/adam.png and /dev/null differ diff --git a/doc/pub/week42/ipynb/figures/rmsprop.png b/doc/pub/week42/ipynb/figures/rmsprop.png deleted file mode 100644 index 9f336d033..000000000 Binary files a/doc/pub/week42/ipynb/figures/rmsprop.png and /dev/null differ diff --git a/doc/pub/week42/ipynb/figures/simplenn3.png b/doc/pub/week42/ipynb/figures/simplenn3.png deleted file mode 100644 index 1f1d25c42..000000000 Binary files a/doc/pub/week42/ipynb/figures/simplenn3.png and /dev/null differ diff --git a/doc/pub/week42/ipynb/week42.ipynb b/doc/pub/week42/ipynb/week42.ipynb index 8699e7c79..6437ae25e 100644 --- a/doc/pub/week42/ipynb/week42.ipynb +++ b/doc/pub/week42/ipynb/week42.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "474b773e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "a8ac8249", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 42 Constructing a Neural Network code with examples\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -28,9 +24,7 @@ { "cell_type": "markdown", "id": "c8760378", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lecture October 14, 2024\n", "1. Building our own Feed-forward Neural Network and discussion of project 2\n", @@ -57,9 +51,7 @@ { "cell_type": "markdown", "id": "76c6895d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Material for the active learning sessions on Tuesday and Wednesday\n", " * Exercise on starting to write a code for neural networks, feed forward part. We will also continue ur discussions of gradient descent methods from last week. If you have time, start considering the back-propagation part as well (exercises for next week)\n", @@ -74,9 +66,7 @@ { "cell_type": "markdown", "id": "ebb39354", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Writing a code which implements a feed-forward neural network\n", "\n", @@ -93,9 +83,7 @@ { "cell_type": "markdown", "id": "cfb26b1b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Mathematics of deep learning\n", "\n", @@ -109,9 +97,7 @@ { "cell_type": "markdown", "id": "28f60678", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Reminder on books with hands-on material and codes\n", "* [Sebastian Rashcka et al, Machine learning with Sickit-Learn and PyTorch](https://sebastianraschka.com/blog/2022/ml-pytorch-book.html)" @@ -120,9 +106,7 @@ { "cell_type": "markdown", "id": "3c7f8f61", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Reading recommendations\n", "\n", @@ -134,9 +118,7 @@ { "cell_type": "markdown", "id": "9ee5faa5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## First network example, simple percepetron with one input\n", "\n", @@ -149,9 +131,7 @@ { "cell_type": "markdown", "id": "e82fbb01", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_1 = w_1x+b_1,\n", @@ -161,9 +141,7 @@ { "cell_type": "markdown", "id": "e1b4862d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $w_1$ is the weight and $b_1$ is the bias. These are the\n", "parameters we want to optimize. The output is $a_1=\\sigma(z_1)$ (see\n", @@ -175,9 +153,7 @@ { "cell_type": "markdown", "id": "f7b41bcb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(x;w_1,b_1)=\\frac{1}{2}(a_1-y)^2.\n", @@ -187,9 +163,7 @@ { "cell_type": "markdown", "id": "2825a689", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Layout of a simple neural network with no hidden layer\n", "\n", @@ -203,9 +177,7 @@ { "cell_type": "markdown", "id": "1853d23c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimizing the parameters\n", "\n", @@ -219,9 +191,7 @@ { "cell_type": "markdown", "id": "1625a8c3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_1} \\hspace{0.1cm}\\mathrm{and}\\hspace{0.1cm}\\frac{\\partial C}{\\partial b_1}.\n", @@ -231,9 +201,7 @@ { "cell_type": "markdown", "id": "7427a24a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using the chain rule we find" ] @@ -241,9 +209,7 @@ { "cell_type": "markdown", "id": "d4911a64", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_1-y)\\sigma_1'x,\n", @@ -253,9 +219,7 @@ { "cell_type": "markdown", "id": "0f290988", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -263,9 +227,7 @@ { "cell_type": "markdown", "id": "4cbbad95", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_1-y)\\sigma_1',\n", @@ -275,9 +237,7 @@ { "cell_type": "markdown", "id": "cc8122a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we later will just define as" ] @@ -285,9 +245,7 @@ { "cell_type": "markdown", "id": "39f03825", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}=\\delta_1.\n", @@ -297,9 +255,7 @@ { "cell_type": "markdown", "id": "3eeb7a29", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Adding a hidden layer\n", "\n", @@ -313,9 +269,7 @@ { "cell_type": "markdown", "id": "33e1d35f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_1 = w_1a_0+b_1 \\hspace{0.1cm} \\wedge a_1 = \\sigma_1(z_1),\n", @@ -325,9 +279,7 @@ { "cell_type": "markdown", "id": "e846a441", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_2 = w_2a_1+b_2 \\hspace{0.1cm} \\wedge a_2 = \\sigma_2(z_2),\n", @@ -337,9 +289,7 @@ { "cell_type": "markdown", "id": "fbb3f1ff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and the cost function" ] @@ -347,9 +297,7 @@ { "cell_type": "markdown", "id": "40200643", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(x;\\boldsymbol{\\Theta})=\\frac{1}{2}(a_2-y)^2,\n", @@ -359,9 +307,7 @@ { "cell_type": "markdown", "id": "a2e96689", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\boldsymbol{\\Theta}=[w_1,w_2,b_1,b_2]$." ] @@ -369,9 +315,7 @@ { "cell_type": "markdown", "id": "1a014316", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Layout of a simple neural network with one hidden layer\n", "\n", @@ -385,9 +329,7 @@ { "cell_type": "markdown", "id": "06a0b720", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivatives\n", "\n", @@ -397,9 +339,7 @@ { "cell_type": "markdown", "id": "48581ce4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial w_2}=(a_2-y)\\sigma_2'a_1=\\delta_2a_1,\n", @@ -409,9 +349,7 @@ { "cell_type": "markdown", "id": "58f46792", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial b_2}=(a_2-y)\\sigma_2'=\\delta_2,\n", @@ -421,9 +359,7 @@ { "cell_type": "markdown", "id": "dc5aa924", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_2-y)\\sigma_2'a_1\\sigma_1'a_0,\n", @@ -433,9 +369,7 @@ { "cell_type": "markdown", "id": "407690af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_2-y)\\sigma_2'\\sigma_1'=\\delta_1.\n", @@ -445,9 +379,7 @@ { "cell_type": "markdown", "id": "8f5cacad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Can you generalize this to more than one hidden layer?" ] @@ -455,9 +387,7 @@ { "cell_type": "markdown", "id": "79f592c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Important observations\n", "\n", @@ -471,9 +401,7 @@ { "cell_type": "markdown", "id": "412247f9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The training\n", "\n", @@ -483,9 +411,7 @@ { "cell_type": "markdown", "id": "83c09361", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{i}\\leftarrow w_{i}- \\eta \\delta_i a_{i-1},\n", @@ -495,9 +421,7 @@ { "cell_type": "markdown", "id": "d341a3c7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -505,9 +429,7 @@ { "cell_type": "markdown", "id": "2bb5b78f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_i \\leftarrow b_i-\\eta \\delta_i,\n", @@ -517,9 +439,7 @@ { "cell_type": "markdown", "id": "3d6558a8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\eta$ is the learning rate.\n", "\n", @@ -531,9 +451,7 @@ { "cell_type": "markdown", "id": "f55c404e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code example\n", "\n", @@ -549,10 +467,7 @@ "cell_type": "code", "execution_count": 1, "id": "076950d8", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -622,9 +537,7 @@ { "cell_type": "markdown", "id": "b5a2150f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small." ] @@ -632,9 +545,7 @@ { "cell_type": "markdown", "id": "cb93f2d2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple neural network and the back propagation equations\n", "\n", @@ -649,9 +560,7 @@ { "cell_type": "markdown", "id": "893b1b33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "x_0 = a_0^{(0)} \\wedge x_1 = a_1^{(0)}.\n", @@ -661,9 +570,7 @@ { "cell_type": "markdown", "id": "f1554902", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_0^{(1)}$ and $a_1^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters" ] @@ -671,9 +578,7 @@ { "cell_type": "markdown", "id": "e01cd011", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{ij}^{(1)}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_0^{(1)},b_1^{(1)}\\right\\}.\n", @@ -683,9 +588,7 @@ { "cell_type": "markdown", "id": "691e42c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Layout of a simple neural network with two input nodes, one hidden layer and one output node\n", "\n", @@ -699,9 +602,7 @@ { "cell_type": "markdown", "id": "57573a8a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The ouput layer\n", "\n", @@ -711,9 +612,7 @@ { "cell_type": "markdown", "id": "91ff8dfb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{i}^{(2)}=\\left\\{w_{0}^{(2)},w_{1}^{(2)}\\right\\} \\wedge b^{(2)}.\n", @@ -723,9 +622,7 @@ { "cell_type": "markdown", "id": "315f787f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our output is $\\tilde{y}=a^{(2)}$ and we define a generic cost function $C(a^{(2)},y;\\boldsymbol{\\Theta})$ where $y$ is the target value (a scalar here).\n", "The parameters we need to optimize are given by" @@ -734,9 +631,7 @@ { "cell_type": "markdown", "id": "612ebd67", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\Theta}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)},w_{0}^{(2)},w_{1}^{(2)},b_0^{(1)},b_1^{(1)},b^{(2)}\\right\\}.\n", @@ -746,9 +641,7 @@ { "cell_type": "markdown", "id": "c28762dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Compact expressions\n", "\n", @@ -759,9 +652,7 @@ { "cell_type": "markdown", "id": "b4d0309d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{bmatrix}z_0^{(1)} \\\\ z_1^{(1)} \\end{bmatrix}=\\left(\\begin{bmatrix}w_{00}^{(1)} & w_{01}^{(1)}\\\\ w_{10}^{(1)} &w_{11}^{(1)} \\end{bmatrix}\\right)^{T}\\begin{bmatrix}a_0^{(0)} \\\\ a_1^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_0^{(1)} \\\\ b_1^{(1)} \\end{bmatrix},\n", @@ -771,9 +662,7 @@ { "cell_type": "markdown", "id": "f3c715f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with outputs" ] @@ -781,9 +670,7 @@ { "cell_type": "markdown", "id": "5db82289", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{bmatrix}a_0^{(1)} \\\\ a_1^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_0^{(1)}) \\\\ \\sigma^{(1)}(z_1^{(1)}) \\end{bmatrix}.\n", @@ -793,9 +680,7 @@ { "cell_type": "markdown", "id": "b6b2ad78", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Output layer\n", "\n", @@ -805,9 +690,7 @@ { "cell_type": "markdown", "id": "dcefdba1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z^{(2)} = w_{0}^{(2)}a_0^{(1)} +w_{1}^{(2)}a_1^{(1)}+b^{(2)},\n", @@ -817,9 +700,7 @@ { "cell_type": "markdown", "id": "d554c6ce", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "resulting in the output" ] @@ -827,9 +708,7 @@ { "cell_type": "markdown", "id": "651ff447", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "a^{(2)}=\\sigma^{(2)}(z^{(2)}).\n", @@ -839,9 +718,7 @@ { "cell_type": "markdown", "id": "f171ceaa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Explicit derivatives\n", "\n", @@ -855,9 +732,7 @@ { "cell_type": "markdown", "id": "609bb2dd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{i}^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial w_{i}^{(2)}}=\\delta^{(2)}a_i^{(1)},\n", @@ -867,9 +742,7 @@ { "cell_type": "markdown", "id": "da1d696b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with" ] @@ -877,9 +750,7 @@ { "cell_type": "markdown", "id": "d21d62cc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta^{(2)}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", @@ -889,9 +760,7 @@ { "cell_type": "markdown", "id": "fb9dd90f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and finally" ] @@ -899,9 +768,7 @@ { "cell_type": "markdown", "id": "9026fb2d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial b^{(2)}}=\\delta^{(2)}.\n", @@ -911,9 +778,7 @@ { "cell_type": "markdown", "id": "2d20d5d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Derivatives of the hidden layer\n", "\n", @@ -923,9 +788,7 @@ { "cell_type": "markdown", "id": "e6fbb9b5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", @@ -936,9 +799,7 @@ { "cell_type": "markdown", "id": "781a7ac8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which, noting that" ] @@ -946,9 +807,7 @@ { "cell_type": "markdown", "id": "3ee09d8f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z^{(2)} =w_0^{(2)}a_0^{(1)}+w_1^{(2)}a_1^{(1)}+b^{(2)},\n", @@ -958,9 +817,7 @@ { "cell_type": "markdown", "id": "65f44692", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "allows us to rewrite" ] @@ -968,9 +825,7 @@ { "cell_type": "markdown", "id": "0b84b7af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}a_0^{(1)}.\n", @@ -980,9 +835,7 @@ { "cell_type": "markdown", "id": "53c138bd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expression\n", "Defining" @@ -991,9 +844,7 @@ { "cell_type": "markdown", "id": "3f81bb3f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_0^{(1)}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}\\delta^{(2)},\n", @@ -1003,9 +854,7 @@ { "cell_type": "markdown", "id": "eb6783b2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have" ] @@ -1013,9 +862,7 @@ { "cell_type": "markdown", "id": "1c754dbb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\delta_0^{(1)}a_0^{(1)}.\n", @@ -1025,9 +872,7 @@ { "cell_type": "markdown", "id": "e0b9760c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Similarly, we obtain" ] @@ -1035,9 +880,7 @@ { "cell_type": "markdown", "id": "b1efb446", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{01}^{(1)}}=\\delta_0^{(1)}a_1^{(1)}.\n", @@ -1047,9 +890,7 @@ { "cell_type": "markdown", "id": "2cf7b43f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Completing the list\n", "\n", @@ -1059,9 +900,7 @@ { "cell_type": "markdown", "id": "262b517f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{10}^{(1)}}=\\delta_1^{(1)}a_0^{(1)},\n", @@ -1071,9 +910,7 @@ { "cell_type": "markdown", "id": "59285e3f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1081,9 +918,7 @@ { "cell_type": "markdown", "id": "2813790f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)},\n", @@ -1093,9 +928,7 @@ { "cell_type": "markdown", "id": "a56b94b4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined" ] @@ -1103,9 +936,7 @@ { "cell_type": "markdown", "id": "a3ed9010", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)}.\n", @@ -1115,9 +946,7 @@ { "cell_type": "markdown", "id": "03a5562c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expressions for the biases of the hidden layer\n", "\n", @@ -1127,9 +956,7 @@ { "cell_type": "markdown", "id": "025bd04f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_{0}^{(1)}}=\\delta_0^{(1)},\n", @@ -1139,9 +966,7 @@ { "cell_type": "markdown", "id": "e1ef6c2f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1149,9 +974,7 @@ { "cell_type": "markdown", "id": "157d7580", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)}.\n", @@ -1161,9 +984,7 @@ { "cell_type": "markdown", "id": "389a7e2d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "As we will see below, these expressions can be generalized in a more compact form." ] @@ -1171,9 +992,7 @@ { "cell_type": "markdown", "id": "bccf9918", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient expressions\n", "\n", @@ -1184,9 +1003,7 @@ { "cell_type": "markdown", "id": "ab05945c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{i}^{(2)}\\leftarrow w_{i}^{(2)}- \\eta \\delta^{(2)} a_{i}^{(1)},\n", @@ -1196,9 +1013,7 @@ { "cell_type": "markdown", "id": "cb2ef195", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1206,9 +1021,7 @@ { "cell_type": "markdown", "id": "da672f61", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b^{(2)} \\leftarrow b^{(2)}-\\eta \\delta^{(2)},\n", @@ -1218,9 +1031,7 @@ { "cell_type": "markdown", "id": "95dd3d2b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1228,9 +1039,7 @@ { "cell_type": "markdown", "id": "532b99a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{ij}^{(1)}\\leftarrow w_{ij}^{(1)}- \\eta \\delta_{i}^{(1)} a_{j}^{(0)},\n", @@ -1240,9 +1049,7 @@ { "cell_type": "markdown", "id": "5d5f45cc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1250,9 +1057,7 @@ { "cell_type": "markdown", "id": "2333cfb1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_{i}^{(1)} \\leftarrow b_{i}^{(1)}-\\eta \\delta_{i}^{(1)},\n", @@ -1262,9 +1067,7 @@ { "cell_type": "markdown", "id": "4b555f81", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\eta$ is the learning rate." ] @@ -1272,9 +1075,7 @@ { "cell_type": "markdown", "id": "20c0952f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the equations for a neural network\n", "\n", @@ -1289,9 +1090,7 @@ { "cell_type": "markdown", "id": "5a155f81", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\cal C}(\\boldsymbol{\\Theta}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2,\n", @@ -1301,9 +1100,7 @@ { "cell_type": "markdown", "id": "ff1fc071", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the $y_i$s are our $n$ targets (the values we want to\n", "reproduce), while the outputs of the network after having propagated\n", @@ -1313,9 +1110,7 @@ { "cell_type": "markdown", "id": "63576478", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Layout of a neural network with three hidden layers (last later = $l=L=4$, first layer $l=0$)\n", "\n", @@ -1329,9 +1124,7 @@ { "cell_type": "markdown", "id": "3410ad89", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Definitions\n", "\n", @@ -1346,9 +1139,7 @@ { "cell_type": "markdown", "id": "c71d887c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_j^l = \\sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,\n", @@ -1358,9 +1149,7 @@ { "cell_type": "markdown", "id": "ffeae21c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $b_k^l$ are the biases from layer $l$. Here $M_{l-1}$\n", "represents the total number of nodes/neurons/units of layer $l-1$. The\n", @@ -1371,9 +1160,7 @@ { "cell_type": "markdown", "id": "2fe917a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{z}^l = \\left(\\boldsymbol{W}^l\\right)^T\\boldsymbol{a}^{l-1}+\\boldsymbol{b}^l.\n", @@ -1383,9 +1170,7 @@ { "cell_type": "markdown", "id": "65404bec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Inputs to the activation function\n", "\n", @@ -1399,9 +1184,7 @@ { "cell_type": "markdown", "id": "a6c0c8b6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "a_j^l = \\sigma(z_j^l) = \\frac{1}{1+\\exp{-(z_j^l)}}.\n", @@ -1411,9 +1194,7 @@ { "cell_type": "markdown", "id": "2454faec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Layout of input to first hidden layer $l=1$ from input layer $l=0$\n", "\n", @@ -1427,9 +1208,7 @@ { "cell_type": "markdown", "id": "e5b67040", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Derivatives and the chain rule\n", "\n", @@ -1439,9 +1218,7 @@ { "cell_type": "markdown", "id": "aaec49a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial z_j^l}{\\partial w_{ij}^l} = a_i^{l-1},\n", @@ -1451,9 +1228,7 @@ { "cell_type": "markdown", "id": "e8ad40d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1461,9 +1236,7 @@ { "cell_type": "markdown", "id": "e04a4d28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial z_j^l}{\\partial a_i^{l-1}} = w_{ji}^l.\n", @@ -1473,9 +1246,7 @@ { "cell_type": "markdown", "id": "9b121a33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "With our definition of the activation function we have that (note that this function depends only on $z_j^l$)" ] @@ -1483,9 +1254,7 @@ { "cell_type": "markdown", "id": "3bf7d884", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial a_j^l}{\\partial z_j^{l}} = a_j^l(1-a_j^l)=\\sigma(z_j^l)(1-\\sigma(z_j^l)).\n", @@ -1495,9 +1264,7 @@ { "cell_type": "markdown", "id": "7d58918c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Derivative of the cost function\n", "\n", @@ -1509,9 +1276,7 @@ { "cell_type": "markdown", "id": "d21dce97", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\cal C}(\\boldsymbol{\\Theta}^L) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2=\\frac{1}{2}\\sum_{i=1}^n\\left(a_i^L - y_i\\right)^2,\n", @@ -1521,9 +1286,7 @@ { "cell_type": "markdown", "id": "dc143bea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The derivative of this function with respect to the weights is" ] @@ -1531,9 +1294,7 @@ { "cell_type": "markdown", "id": "49415a06", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial{\\cal C}(\\boldsymbol{\\Theta}^L)}{\\partial w_{ij}^L} = \\left(a_j^L - y_j\\right)\\frac{\\partial a_j^L}{\\partial w_{ij}^{L}},\n", @@ -1543,9 +1304,7 @@ { "cell_type": "markdown", "id": "4398e11e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The last partial derivative can easily be computed and reads (by applying the chain rule)" ] @@ -1553,9 +1312,7 @@ { "cell_type": "markdown", "id": "da3fca7e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial a_j^L}{\\partial w_{ij}^{L}} = \\frac{\\partial a_j^L}{\\partial z_{j}^{L}}\\frac{\\partial z_j^L}{\\partial w_{ij}^{L}}=a_j^L(1-a_j^L)a_i^{L-1}.\n", @@ -1565,9 +1322,7 @@ { "cell_type": "markdown", "id": "9055bc57", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The back propagation equations for a neural network\n", "\n", @@ -1577,9 +1332,7 @@ { "cell_type": "markdown", "id": "c7995e73", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial{\\cal C}((\\boldsymbol{\\Theta}^L)}{\\partial w_{ij}^L} = \\left(a_j^L - y_j\\right)a_j^L(1-a_j^L)a_i^{L-1},\n", @@ -1589,9 +1342,7 @@ { "cell_type": "markdown", "id": "d142f966", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Defining" ] @@ -1599,9 +1350,7 @@ { "cell_type": "markdown", "id": "d5dc5f9f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L = a_j^L(1-a_j^L)\\left(a_j^L - y_j\\right) = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n", @@ -1611,9 +1360,7 @@ { "cell_type": "markdown", "id": "5f598e3e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and using the Hadamard product of two vectors we can write this as" ] @@ -1621,9 +1368,7 @@ { "cell_type": "markdown", "id": "9769b50e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\delta}^L = \\sigma'(\\boldsymbol{z}^L)\\circ\\frac{\\partial {\\cal C}}{\\partial (\\boldsymbol{a}^L)}.\n", @@ -1633,9 +1378,7 @@ { "cell_type": "markdown", "id": "5a8851ae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Analyzing the last results\n", "\n", @@ -1651,9 +1394,7 @@ { "cell_type": "markdown", "id": "b455efff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More considerations\n", "\n", @@ -1669,9 +1410,7 @@ { "cell_type": "markdown", "id": "8b430778", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}\n", @@ -1681,9 +1420,7 @@ { "cell_type": "markdown", "id": "7fb2cea5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "With the definition of $\\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely" ] @@ -1691,9 +1428,7 @@ { "cell_type": "markdown", "id": "6c77ac74", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial{\\cal C}}{\\partial w_{ij}^L} = \\delta_j^La_i^{L-1}.\n", @@ -1703,9 +1438,7 @@ { "cell_type": "markdown", "id": "708f42bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Derivatives in terms of $z_j^L$\n", "\n", @@ -1715,9 +1448,7 @@ { "cell_type": "markdown", "id": "40f66bf5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L =\\frac{\\partial {\\cal C}}{\\partial z_j^L}= \\frac{\\partial {\\cal C}}{\\partial a_j^L}\\frac{\\partial a_j^L}{\\partial z_j^L},\n", @@ -1727,9 +1458,7 @@ { "cell_type": "markdown", "id": "21b9ebbe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely" ] @@ -1737,9 +1466,7 @@ { "cell_type": "markdown", "id": "665ad548", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L}\\frac{\\partial b_j^L}{\\partial z_j^L}=\\frac{\\partial {\\cal C}}{\\partial b_j^L},\n", @@ -1749,9 +1476,7 @@ { "cell_type": "markdown", "id": "c182abd9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "That is, the error $\\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias." ] @@ -1759,9 +1484,7 @@ { "cell_type": "markdown", "id": "aa70c6cf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bringing it together\n", "\n", @@ -1771,9 +1494,7 @@ { "cell_type": "markdown", "id": "db460a07", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -1789,9 +1510,7 @@ { "cell_type": "markdown", "id": "3af036d6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1799,9 +1518,7 @@ { "cell_type": "markdown", "id": "5662288a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -1817,9 +1534,7 @@ { "cell_type": "markdown", "id": "b6a8a1ac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1827,9 +1542,7 @@ { "cell_type": "markdown", "id": "dda8fce0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -1845,9 +1558,7 @@ { "cell_type": "markdown", "id": "f1957ab0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final back propagating equation\n", "\n", @@ -1857,9 +1568,7 @@ { "cell_type": "markdown", "id": "31c8f2c9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l =\\frac{\\partial {\\cal C}}{\\partial z_j^l}.\n", @@ -1869,9 +1578,7 @@ { "cell_type": "markdown", "id": "bba8d29e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We want to express this in terms of the equations for layer $l+1$." ] @@ -1879,9 +1586,7 @@ { "cell_type": "markdown", "id": "af4922ec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using the chain rule and summing over all $k$ entries\n", "\n", @@ -1891,9 +1596,7 @@ { "cell_type": "markdown", "id": "f43f0f87", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l =\\sum_k \\frac{\\partial {\\cal C}}{\\partial z_k^{l+1}}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}}=\\sum_k \\delta_k^{l+1}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}},\n", @@ -1903,9 +1606,7 @@ { "cell_type": "markdown", "id": "eba489e4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and recalling that" ] @@ -1913,9 +1614,7 @@ { "cell_type": "markdown", "id": "82610f92", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_j^{l+1} = \\sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},\n", @@ -1925,9 +1624,7 @@ { "cell_type": "markdown", "id": "62513d62", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $M_l$ being the number of nodes in layer $l$, we obtain" ] @@ -1935,9 +1632,7 @@ { "cell_type": "markdown", "id": "ca381ec8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l =\\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", @@ -1947,9 +1642,7 @@ { "cell_type": "markdown", "id": "bdf32ad4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This is our final equation.\n", "\n", @@ -1959,9 +1652,7 @@ { "cell_type": "markdown", "id": "764448ea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the back propagation algorithm\n", "\n", @@ -1982,9 +1673,7 @@ { "cell_type": "markdown", "id": "989c3082", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the back propagation algorithm, part 2\n", "\n", @@ -1994,9 +1683,7 @@ { "cell_type": "markdown", "id": "9d14949a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n", @@ -2006,9 +1693,7 @@ { "cell_type": "markdown", "id": "4ee118b1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Then we compute the back propagate error for each $l=L-1,L-2,\\dots,1$ as" ] @@ -2016,9 +1701,7 @@ { "cell_type": "markdown", "id": "ba89ad67", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n", @@ -2028,9 +1711,7 @@ { "cell_type": "markdown", "id": "e1f840c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the Back propagation algorithm, part 3\n", "\n", @@ -2042,9 +1723,7 @@ { "cell_type": "markdown", "id": "9f06bcfc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", @@ -2054,9 +1733,7 @@ { "cell_type": "markdown", "id": "7063f2be", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", @@ -2066,9 +1743,7 @@ { "cell_type": "markdown", "id": "3e299445", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\eta$ being the learning rate." ] @@ -2076,9 +1751,7 @@ { "cell_type": "markdown", "id": "28921878", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Updating the gradients\n", "\n", @@ -2088,9 +1761,7 @@ { "cell_type": "markdown", "id": "90f5404c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", @@ -2100,9 +1771,7 @@ { "cell_type": "markdown", "id": "53c2dfbd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,1$ and update the weights and biases according to the rules" ] @@ -2110,9 +1779,7 @@ { "cell_type": "markdown", "id": "03805799", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", @@ -2122,9 +1789,7 @@ { "cell_type": "markdown", "id": "4f8f9c02", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", @@ -2134,9 +1799,7 @@ { "cell_type": "markdown", "id": "9bf86f9c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Activation functions\n", "\n", @@ -2157,9 +1820,7 @@ { "cell_type": "markdown", "id": "eeaed73c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Activation functions, Logistic and Hyperbolic ones\n", "\n", @@ -2176,9 +1837,7 @@ { "cell_type": "markdown", "id": "0786aaf0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sigma(x) = \\frac{1}{1 + e^{-x}},\n", @@ -2188,9 +1847,7 @@ { "cell_type": "markdown", "id": "aa76601b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and the *hyperbolic tangent* function" ] @@ -2198,9 +1855,7 @@ { "cell_type": "markdown", "id": "e1bca915", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sigma(x) = \\tanh(x)\n", @@ -2210,9 +1865,7 @@ { "cell_type": "markdown", "id": "1832a8c4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Relevance\n", "\n", @@ -2227,10 +1880,7 @@ "cell_type": "code", "execution_count": 2, "id": "5c689736", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -2311,9 +1961,7 @@ { "cell_type": "markdown", "id": "0b3a4c95", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Fine-tuning neural network hyperparameters\n", "\n", @@ -2340,9 +1988,7 @@ { "cell_type": "markdown", "id": "9955a636", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Hidden layers\n", "\n", @@ -2366,9 +2012,7 @@ { "cell_type": "markdown", "id": "f1aa77e9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Vanishing gradients\n", "\n", @@ -2388,9 +2032,7 @@ { "cell_type": "markdown", "id": "3703bb33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Exploding gradients\n", "\n", @@ -2406,9 +2048,7 @@ { "cell_type": "markdown", "id": "43b1aa26", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Is the Logistic activation function (Sigmoid) our choice?\n", "\n", @@ -2429,9 +2069,7 @@ { "cell_type": "markdown", "id": "463f4f64", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Logistic function as the root of problems\n", "\n", @@ -2448,9 +2086,7 @@ { "cell_type": "markdown", "id": "6c9ea582", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivative of the Logistic funtion\n", "\n", @@ -2476,9 +2112,7 @@ { "cell_type": "markdown", "id": "80c83d2c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Insights from the paper by Glorot and Bengio\n", "\n", @@ -2496,9 +2130,7 @@ { "cell_type": "markdown", "id": "55a1ad5d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The RELU function family\n", "\n", @@ -2517,9 +2149,7 @@ { "cell_type": "markdown", "id": "b7fafe4a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## ELU function\n", "\n", @@ -2531,9 +2161,7 @@ { "cell_type": "markdown", "id": "46e10153", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", @@ -2543,9 +2171,7 @@ { "cell_type": "markdown", "id": "0930ba50", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Which activation function should we use?\n", "\n", @@ -2565,9 +2191,7 @@ { "cell_type": "markdown", "id": "5c0c59af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on activation functions, output layers\n", "\n", @@ -2587,9 +2211,7 @@ { "cell_type": "markdown", "id": "582e4df4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Batch Normalization\n", "\n", @@ -2613,9 +2235,7 @@ { "cell_type": "markdown", "id": "ea360013", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Dropout\n", "\n", @@ -2633,9 +2253,7 @@ { "cell_type": "markdown", "id": "fee98b12", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient Clipping\n", "\n", @@ -2653,9 +2271,7 @@ { "cell_type": "markdown", "id": "c9f175b0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A top-down perspective on Neural networks\n", "\n", @@ -2679,9 +2295,7 @@ { "cell_type": "markdown", "id": "f6f04fbb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More top-down perspectives\n", "\n", @@ -2708,9 +2322,7 @@ { "cell_type": "markdown", "id": "d4184eee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Limitations of supervised learning with deep networks\n", "\n", @@ -2726,9 +2338,7 @@ { "cell_type": "markdown", "id": "82d314da", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Limitations of NNs\n", "\n", @@ -2742,9 +2352,7 @@ { "cell_type": "markdown", "id": "abc6640b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Homogeneous data\n", "\n", @@ -2754,9 +2362,7 @@ { "cell_type": "markdown", "id": "bb7d1f89", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More limitations\n", "\n", @@ -2768,9 +2374,7 @@ { "cell_type": "markdown", "id": "442b1b5e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the back-propagation algorithm\n", "\n", @@ -2791,9 +2395,7 @@ { "cell_type": "markdown", "id": "0005e340", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n", @@ -2803,9 +2405,7 @@ { "cell_type": "markdown", "id": "2b27751b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as" ] @@ -2813,9 +2413,7 @@ { "cell_type": "markdown", "id": "dd544b4b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n", @@ -2825,9 +2423,7 @@ { "cell_type": "markdown", "id": "57a7d213", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules" ] @@ -2835,9 +2431,7 @@ { "cell_type": "markdown", "id": "932510f4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", @@ -2847,9 +2441,7 @@ { "cell_type": "markdown", "id": "bf80ed99", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", @@ -2859,9 +2451,7 @@ { "cell_type": "markdown", "id": "f682bf76", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n", "Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training." @@ -2870,9 +2460,7 @@ { "cell_type": "markdown", "id": "ad2b26a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up a Multi-layer perceptron model for classification\n", "\n", @@ -2898,9 +2486,7 @@ { "cell_type": "markdown", "id": "a08331b2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp{(- \\boldsymbol{x}})} ,\n", @@ -2910,9 +2496,7 @@ { "cell_type": "markdown", "id": "9ebfde52", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -2920,9 +2504,7 @@ { "cell_type": "markdown", "id": "531dfa25", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "P(y = 1 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = 1 - P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) ,\n", @@ -2932,9 +2514,7 @@ { "cell_type": "markdown", "id": "8941d38c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $y \\in \\{0, 1\\}$ and $\\boldsymbol{\\theta}$ represents the weights and biases\n", "of our network." @@ -2943,9 +2523,7 @@ { "cell_type": "markdown", "id": "7ae7e222", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Defining the cost function\n", "\n", @@ -2955,9 +2533,7 @@ { "cell_type": "markdown", "id": "cbb157cc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\theta}) = - \\ln P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = - \\sum_{i=1}^n\n", @@ -2968,9 +2544,7 @@ { "cell_type": "markdown", "id": "0178770c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This last equality means that we can interpret our *cost* function as a sum over the *loss* function\n", "for each point in the dataset $\\mathcal{L}_i(\\boldsymbol{\\theta})$. \n", @@ -2993,9 +2567,7 @@ { "cell_type": "markdown", "id": "cb69f3df", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "P(y_{ic} = 1 \\mid \\boldsymbol{x}_i, \\boldsymbol{\\theta}) = \\frac{\\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_c)}}\n", @@ -3006,9 +2578,7 @@ { "cell_type": "markdown", "id": "11aa1335", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which reduces to the logistic function in the binary case. \n", "The likelihood of this $C$-class classifier\n", @@ -3018,9 +2588,7 @@ { "cell_type": "markdown", "id": "a225e57a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = \\prod_{i=1}^n \\prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} .\n", @@ -3030,9 +2598,7 @@ { "cell_type": "markdown", "id": "569dd3f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Again we take the negative log-likelihood to define our cost function:" ] @@ -3040,9 +2606,7 @@ { "cell_type": "markdown", "id": "f8e84cae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\theta}) = - \\log{P(\\mathcal{D} \\mid \\boldsymbol{\\theta})}.\n", @@ -3052,9 +2616,7 @@ { "cell_type": "markdown", "id": "1dd303e8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "See the logistic regression lectures for a full definition of the cost function.\n", "\n", @@ -3064,9 +2626,7 @@ { "cell_type": "markdown", "id": "33494270", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Example: binary classification problem\n", "\n", @@ -3076,9 +2636,7 @@ { "cell_type": "markdown", "id": "01bd6a05", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta}) = - \\sum_{i=1}^n \\left(y_i\\log{p(y_i \\vert x_i,\\boldsymbol{\\beta})}+(1-y_i)\\log{1-p(y_i \\vert x_i,\\boldsymbol{\\beta})}\\right),\n", @@ -3088,9 +2646,7 @@ { "cell_type": "markdown", "id": "60f16a47", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we had defined the logistic (sigmoid) function" ] @@ -3098,9 +2654,7 @@ { "cell_type": "markdown", "id": "ab0c42b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(y_i =1\\vert x_i,\\boldsymbol{\\beta})=\\frac{\\exp{(\\beta_0+\\beta_1 x_i)}}{1+\\exp{(\\beta_0+\\beta_1 x_i)}},\n", @@ -3110,9 +2664,7 @@ { "cell_type": "markdown", "id": "dcfb2e28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -3120,9 +2672,7 @@ { "cell_type": "markdown", "id": "5076b392", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(y_i =0\\vert x_i,\\boldsymbol{\\beta})=1-p(y_i =1\\vert x_i,\\boldsymbol{\\beta}).\n", @@ -3132,9 +2682,7 @@ { "cell_type": "markdown", "id": "9d6001f4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The parameters $\\boldsymbol{\\beta}$ were defined using a minimization method like gradient descent or Newton-Raphson's method. \n", "\n", @@ -3145,9 +2693,7 @@ { "cell_type": "markdown", "id": "d0245c18", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "a_i^l = y_i = \\frac{\\exp{(z_i^l)}}{1+\\exp{(z_i^l)}},\n", @@ -3157,9 +2703,7 @@ { "cell_type": "markdown", "id": "030ac014", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with" ] @@ -3167,9 +2711,7 @@ { "cell_type": "markdown", "id": "bdc01c83", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_i^l = \\sum_{j}w_{ij}^l a_j^{l-1}+b_i^l,\n", @@ -3179,9 +2721,7 @@ { "cell_type": "markdown", "id": "1a77b25c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the superscript $l-1$ indicates that these are the outputs from layer $l-1$.\n", "Our cost function at the final layer $l=L$ is now" @@ -3190,9 +2730,7 @@ { "cell_type": "markdown", "id": "1f169ed1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{W}) = - \\sum_{i=1}^n \\left(t_i\\log{a_i^L}+(1-t_i)\\log{(1-a_i^L)}\\right),\n", @@ -3202,9 +2740,7 @@ { "cell_type": "markdown", "id": "6bc460e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined the targets $t_i$. The derivatives of the cost function with respect to the output $a_i^L$ are then easily calculated and we get" ] @@ -3212,9 +2748,7 @@ { "cell_type": "markdown", "id": "fe1f92c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{W})}{\\partial a_i^L} = \\frac{a_i^L-t_i}{a_i^L(1-a_i^L)}.\n", @@ -3224,9 +2758,7 @@ { "cell_type": "markdown", "id": "78cf6843", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In case we use another activation function than the logistic one, we need to evaluate other derivatives." ] @@ -3234,9 +2766,7 @@ { "cell_type": "markdown", "id": "81ed1ccc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Softmax function\n", "In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need" @@ -3245,9 +2775,7 @@ { "cell_type": "markdown", "id": "2a16cf53", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial f(z_i^l)}{\\partial w_{jk}^l} =\n", @@ -3258,9 +2786,7 @@ { "cell_type": "markdown", "id": "21988141", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For the Softmax function we have" ] @@ -3268,9 +2794,7 @@ { "cell_type": "markdown", "id": "b37cc0cd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(z_i^l) = \\frac{\\exp{(z_i^l)}}{\\sum_{m=1}^K\\exp{(z_m^l)}}.\n", @@ -3280,9 +2804,7 @@ { "cell_type": "markdown", "id": "259618c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Its derivative with respect to $z_j^l$ gives" ] @@ -3290,9 +2812,7 @@ { "cell_type": "markdown", "id": "af5a331e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial f(z_i^l)}{\\partial z_j^l}= f(z_i^l)\\left(\\delta_{ij}-f(z_j^l)\\right),\n", @@ -3302,9 +2822,7 @@ { "cell_type": "markdown", "id": "208ab286", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which in case of the simply binary model reduces to having $i=j$." ] @@ -3312,9 +2830,7 @@ { "cell_type": "markdown", "id": "8f30fbc3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Developing a code for doing neural networks with back propagation\n", "\n", @@ -3336,9 +2852,7 @@ { "cell_type": "markdown", "id": "09538ef9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Collect and pre-process data\n", "\n", @@ -3386,10 +2900,7 @@ "cell_type": "code", "execution_count": 3, "id": "ced37e42", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# import necessary packages\n", @@ -3439,9 +2950,7 @@ { "cell_type": "markdown", "id": "ebabb274", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Train and test datasets\n", "\n", @@ -3460,10 +2969,7 @@ "cell_type": "code", "execution_count": 4, "id": "2c87b9f9", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", @@ -3498,9 +3004,7 @@ { "cell_type": "markdown", "id": "0c186fec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Define model and architecture\n", "\n", @@ -3542,9 +3046,7 @@ { "cell_type": "markdown", "id": "2cbece98", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Layers\n", "\n", @@ -3582,9 +3084,7 @@ { "cell_type": "markdown", "id": "442935af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Weights and biases\n", "\n", @@ -3603,10 +3103,7 @@ "cell_type": "code", "execution_count": 5, "id": "07fbb18c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# building our neural network\n", @@ -3629,9 +3126,7 @@ { "cell_type": "markdown", "id": "6a3ea326", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Feed-forward pass\n", "\n", @@ -3657,9 +3152,7 @@ { "cell_type": "markdown", "id": "525624c4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Matrix multiplications\n", "\n", @@ -3694,10 +3187,7 @@ "cell_type": "code", "execution_count": 6, "id": "32d0a43f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# setup the feed-forward pass, subscript h = hidden layer\n", @@ -3740,9 +3230,7 @@ { "cell_type": "markdown", "id": "da2ede88", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Choose cost function and optimizer\n", "\n", @@ -3771,9 +3259,7 @@ { "cell_type": "markdown", "id": "069a250d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimizing the cost function\n", "\n", @@ -3809,9 +3295,7 @@ { "cell_type": "markdown", "id": "29ab6ee7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Regularization\n", "\n", @@ -3843,9 +3327,7 @@ { "cell_type": "markdown", "id": "b7c4a178", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Matrix multiplication\n", "\n", @@ -3884,10 +3366,7 @@ "cell_type": "code", "execution_count": 7, "id": "4142969e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# to categorical turns our integer vector into a onehot representation\n", @@ -3963,9 +3442,7 @@ { "cell_type": "markdown", "id": "4d79c3b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Improving performance\n", "\n", @@ -3984,9 +3461,7 @@ { "cell_type": "markdown", "id": "58012d93", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Full object-oriented implementation\n", "\n", @@ -3998,10 +3473,7 @@ "cell_type": "code", "execution_count": 8, "id": "ea4b7741", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "class NeuralNetwork:\n", @@ -4108,9 +3580,7 @@ { "cell_type": "markdown", "id": "04860b2c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Evaluate model performance on test data\n", "\n", @@ -4127,10 +3597,7 @@ "cell_type": "code", "execution_count": 9, "id": "b52d4df3", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "epochs = 100\n", @@ -4154,9 +3621,7 @@ { "cell_type": "markdown", "id": "6dce68ff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Adjust hyperparameters\n", "\n", @@ -4168,10 +3633,7 @@ "cell_type": "code", "execution_count": 10, "id": "65caeb78", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "eta_vals = np.logspace(-5, 1, 7)\n", @@ -4199,9 +3661,7 @@ { "cell_type": "markdown", "id": "0c44c38e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Visualization" ] @@ -4210,10 +3670,7 @@ "cell_type": "code", "execution_count": 11, "id": "fbdc64f1", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# visual representation of grid search\n", @@ -4254,9 +3711,7 @@ { "cell_type": "markdown", "id": "e07e69e3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## scikit-learn implementation\n", "\n", @@ -4277,10 +3732,7 @@ "cell_type": "code", "execution_count": 12, "id": "f48eaa77", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.neural_network import MLPClassifier\n", @@ -4304,9 +3756,7 @@ { "cell_type": "markdown", "id": "ac3c7bcb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Visualization" ] @@ -4315,10 +3765,7 @@ "cell_type": "code", "execution_count": 13, "id": "2d81262e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# optional\n", @@ -4360,9 +3807,7 @@ { "cell_type": "markdown", "id": "8f5f8f4b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Building neural networks in Tensorflow and Keras\n", "\n", @@ -4378,9 +3823,7 @@ { "cell_type": "markdown", "id": "6ec638b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Tensorflow\n", "\n", @@ -4413,10 +3856,7 @@ "cell_type": "code", "execution_count": 14, "id": "6443ca72", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "pip3 install tensorflow" @@ -4425,9 +3865,7 @@ { "cell_type": "markdown", "id": "a8918d58", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and/or if you use **anaconda**, just write (or install from the graphical user interface)\n", "(current release of CPU-only TensorFlow)" @@ -4437,10 +3875,7 @@ "cell_type": "code", "execution_count": 15, "id": "20e9a0f5", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "conda create -n tf tensorflow\n", @@ -4450,9 +3885,7 @@ { "cell_type": "markdown", "id": "1381a34b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "To install the current release of GPU TensorFlow" ] @@ -4461,10 +3894,7 @@ "cell_type": "code", "execution_count": 16, "id": "80e92af9", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "conda create -n tf-gpu tensorflow-gpu\n", @@ -4474,9 +3904,7 @@ { "cell_type": "markdown", "id": "074ac69f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using Keras\n", "\n", @@ -4489,10 +3917,7 @@ "cell_type": "code", "execution_count": 17, "id": "0d710bd3", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "conda install keras" @@ -4501,9 +3926,7 @@ { "cell_type": "markdown", "id": "7adbcab8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "You can look up the [instructions here](https://keras.io/) for more information.\n", "\n", @@ -4513,9 +3936,7 @@ { "cell_type": "markdown", "id": "e17c7253", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Collect and pre-process data\n", "\n", @@ -4526,10 +3947,7 @@ "cell_type": "code", "execution_count": 18, "id": "5ae741f5", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# import necessary packages\n", @@ -4581,10 +3999,7 @@ "cell_type": "code", "execution_count": 19, "id": "4f7ef6bb", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from tensorflow.keras.layers import Input\n", @@ -4610,10 +4025,7 @@ "cell_type": "code", "execution_count": 20, "id": "7b74e049", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -4640,10 +4052,7 @@ "cell_type": "code", "execution_count": 21, "id": "eaa25983", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", @@ -4667,10 +4076,7 @@ "cell_type": "code", "execution_count": 22, "id": "949eca1f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# optional\n", @@ -4709,9 +4115,7 @@ { "cell_type": "markdown", "id": "bb8b8ac1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Breast Cancer Data, now with Keras" ] @@ -4720,10 +4124,7 @@ "cell_type": "code", "execution_count": 23, "id": "9d5ebbb2", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -4897,9 +4298,7 @@ { "cell_type": "markdown", "id": "f1baeb0b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Building a neural network code\n", "\n", @@ -4916,9 +4315,7 @@ { "cell_type": "markdown", "id": "6549fa13", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Learning rate methods\n", "\n", @@ -4938,10 +4335,7 @@ "cell_type": "code", "execution_count": 24, "id": "21062ebc", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -5079,9 +4473,7 @@ { "cell_type": "markdown", "id": "f965b277", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Usage of the above learning rate schedulers\n", "\n", @@ -5095,10 +4487,7 @@ "cell_type": "code", "execution_count": 25, "id": "35ab935b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", @@ -5108,9 +4497,7 @@ { "cell_type": "markdown", "id": "b26972b4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here is a small example for how a segment of code using schedulers\n", "could look. Switching out the schedulers is simple." @@ -5120,10 +4507,7 @@ "cell_type": "code", "execution_count": 26, "id": "3df9caac", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "weights = np.ones((3,3))\n", @@ -5142,9 +4526,7 @@ { "cell_type": "markdown", "id": "1c07f6da", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Cost functions\n", "\n", @@ -5158,10 +4540,7 @@ "cell_type": "code", "execution_count": 27, "id": "743aa0bd", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -5196,9 +4575,7 @@ { "cell_type": "markdown", "id": "c78bdc16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Below we give a short example of how these cost function may be used\n", "to obtain results if you wish to test them out on your own using\n", @@ -5209,10 +4586,7 @@ "cell_type": "code", "execution_count": 28, "id": "4ec3a28c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from autograd import grad\n", @@ -5230,9 +4604,7 @@ { "cell_type": "markdown", "id": "5c924e11", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Activation functions\n", "\n", @@ -5246,10 +4618,7 @@ "cell_type": "code", "execution_count": 29, "id": "87402ee9", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -5304,9 +4673,7 @@ { "cell_type": "markdown", "id": "45ddd0bc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Below follows a short demonstration of how to use an activation\n", "function. The derivative of the activation function will be important\n", @@ -5319,10 +4686,7 @@ "cell_type": "code", "execution_count": 30, "id": "77034679", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "z = np.array([[4, 5, 6]]).T\n", @@ -5340,9 +4704,7 @@ { "cell_type": "markdown", "id": "9fd8140f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### The Neural Network\n", "\n", @@ -5364,10 +4726,7 @@ "cell_type": "code", "execution_count": 31, "id": "7540d755", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import math\n", @@ -5836,9 +5195,7 @@ { "cell_type": "markdown", "id": "5ee996ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Before we make a model, we will quickly generate a dataset we can use\n", "for our linear regression problem as shown below" @@ -5848,10 +5205,7 @@ "cell_type": "code", "execution_count": 32, "id": "183b180b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -5892,9 +5246,7 @@ { "cell_type": "markdown", "id": "21a48508", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Now that we have our dataset ready for the regression, we can create\n", "our regressor. Note that with the seed parameter, we can make sure our\n", @@ -5908,10 +5260,7 @@ "cell_type": "code", "execution_count": 33, "id": "37eba90c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -5923,9 +5272,7 @@ { "cell_type": "markdown", "id": "58a6b726", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We then fit our model with our training data using the scheduler of our choice." ] @@ -5934,10 +5281,7 @@ "cell_type": "code", "execution_count": 34, "id": "046d7076", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -5949,9 +5293,7 @@ { "cell_type": "markdown", "id": "981e95ce", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Due to the progress bar we can see the MSE (train_error) throughout\n", "the FFNN's training. Note that the fit() function has some optional\n", @@ -5965,10 +5307,7 @@ "cell_type": "code", "execution_count": 35, "id": "6da23296", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -5979,9 +5318,7 @@ { "cell_type": "markdown", "id": "ba0fea41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", "\n", @@ -5994,10 +5331,7 @@ "cell_type": "code", "execution_count": 36, "id": "1bd5c3e5", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", @@ -6020,10 +5354,7 @@ "cell_type": "code", "execution_count": 37, "id": "9923ffbf", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -6035,9 +5366,7 @@ { "cell_type": "markdown", "id": "95431510", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We will now make use of our validation data by passing it into our fit function as a keyword argument" ] @@ -6046,10 +5375,7 @@ "cell_type": "code", "execution_count": 38, "id": "9d71cef9", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -6061,9 +5387,7 @@ { "cell_type": "markdown", "id": "acfee190", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Finally, we will create a neural network with 2 hidden layers with activation functions." ] @@ -6072,10 +5396,7 @@ "cell_type": "code", "execution_count": 39, "id": "c834fb60", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -6092,10 +5413,7 @@ "cell_type": "code", "execution_count": 40, "id": "0c557301", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -6107,9 +5425,7 @@ { "cell_type": "markdown", "id": "750490a3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Multiclass classification\n", "\n", @@ -6122,10 +5438,7 @@ "cell_type": "code", "execution_count": 41, "id": "5318d06b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_digits\n", @@ -6159,9 +5472,7 @@ { "cell_type": "markdown", "id": "ff519831", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Testing the XOR gate and other gates\n", "\n", @@ -6172,10 +5483,7 @@ "cell_type": "code", "execution_count": 42, "id": "57e3fb33", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", @@ -6195,15 +5503,31 @@ { "cell_type": "markdown", "id": "e4c08bad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Not bad, but the results depend strongly on the learning reate. Try different learning rates." ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.15" + } + }, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week42/figures/simplenn3.png b/doc/src/week42/figures/simplenn3.png deleted file mode 100644 index 1f1d25c42..000000000 Binary files a/doc/src/week42/figures/simplenn3.png and /dev/null differ