diff --git a/doc/pub/week40/ipynb/week40.ipynb b/doc/pub/week40/ipynb/week40.ipynb
index c780098c0..508d24842 100644
--- a/doc/pub/week40/ipynb/week40.ipynb
+++ b/doc/pub/week40/ipynb/week40.ipynb
@@ -3,9 +3,7 @@
{
"cell_type": "markdown",
"id": "13266b2f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
@@ -15,9 +13,7 @@
{
"cell_type": "markdown",
"id": "4d8e0fa9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"# Week 40: Neural networks\n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo, Norway and Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, USA\n",
@@ -30,9 +26,7 @@
{
"cell_type": "markdown",
"id": "a4da605c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Plan for week 40\n",
"\n",
@@ -50,9 +44,7 @@
{
"cell_type": "markdown",
"id": "2e32a23b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Overview video on Stochastic Gradient Descent\n",
"\n",
@@ -62,9 +54,7 @@
{
"cell_type": "markdown",
"id": "7adb6ed2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Practical tips\n",
"\n",
@@ -82,9 +72,7 @@
{
"cell_type": "markdown",
"id": "b4c25c60",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Automatic differentiation\n",
"\n",
@@ -117,9 +105,7 @@
{
"cell_type": "markdown",
"id": "4f91a7a5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Videos on Neural Networks\n",
"\n",
@@ -131,9 +117,7 @@
{
"cell_type": "markdown",
"id": "85bd2aa6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Neural networks\n",
"\n",
@@ -149,9 +133,7 @@
{
"cell_type": "markdown",
"id": "54933062",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Artificial neurons\n",
"\n",
@@ -173,9 +155,7 @@
{
"cell_type": "markdown",
"id": "a779d2b7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -191,9 +171,7 @@
{
"cell_type": "markdown",
"id": "f477871e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Here, the output $y$ of the neuron is the value of its activation function, which have as input\n",
"a weighted sum of signals $x_i, \\dots ,x_n$ received by $n$ other neurons.\n",
@@ -231,9 +209,7 @@
{
"cell_type": "markdown",
"id": "1d45a200",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Neural network types\n",
"\n",
@@ -260,9 +236,7 @@
{
"cell_type": "markdown",
"id": "55692795",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Feed-forward neural networks\n",
"\n",
@@ -281,9 +255,7 @@
{
"cell_type": "markdown",
"id": "eff28146",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Convolutional Neural Network\n",
"\n",
@@ -310,9 +282,7 @@
{
"cell_type": "markdown",
"id": "eb5568d2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Recurrent neural networks\n",
"\n",
@@ -331,9 +301,7 @@
{
"cell_type": "markdown",
"id": "389f085c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Other types of networks\n",
"\n",
@@ -352,9 +320,7 @@
{
"cell_type": "markdown",
"id": "baadf7ca",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Multilayer perceptrons\n",
"\n",
@@ -369,9 +335,7 @@
{
"cell_type": "markdown",
"id": "47bb51ac",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Why multilayer perceptrons?\n",
"\n",
@@ -390,9 +354,7 @@
{
"cell_type": "markdown",
"id": "0bf0ebf0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Illustration of a single perceptropn model and a multi-perceptron model\n",
"\n",
@@ -406,9 +368,7 @@
{
"cell_type": "markdown",
"id": "72681bf1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Examples of XOR, OR and AND gates\n",
"\n",
@@ -423,11 +383,27 @@
"cell_type": "code",
"execution_count": 1,
"id": "ba41d9cb",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "The X.TX matrix:[[4. 2. 2.]\n",
+ " [2. 2. 1.]\n",
+ " [2. 1. 2.]]\n",
+ "The invers of X.TX matrix:[[ 7.50000000e-01 -5.00000000e-01 -5.00000000e-01]\n",
+ " [-5.00000000e-01 1.00000000e+00 -2.22044605e-16]\n",
+ " [-5.00000000e-01 -4.44089210e-16 1.00000000e+00]]\n",
+ "The values of theta for the XOR gate:[ 5.00000000e-01 -1.66533454e-16 -4.44089210e-16]\n",
+ "The linear regression prediction for the XOR gate:[0.5 0.5 0.5 0.5]\n",
+ "The values of theta for the OR gate:[0.25 0.5 0.5 ]\n",
+ "The linear regression prediction for the OR gate:[0.25 0.75 0.75 1.25]\n",
+ "The values of theta for the AND gate:[-0.25 0.5 0.5 ]\n",
+ "The linear regression prediction for the AND gate:[-0.25 0.25 0.25 0.75]\n"
+ ]
+ }
+ ],
"source": [
"\"\"\"\n",
"Simple code that tests XOR, OR and AND gates with linear regression\n",
@@ -464,9 +440,7 @@
{
"cell_type": "markdown",
"id": "1d54a410",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"What is happening here?"
]
@@ -474,9 +448,7 @@
{
"cell_type": "markdown",
"id": "853be7cb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Does Logistic Regression do a better Job?"
]
@@ -485,11 +457,30 @@
"cell_type": "code",
"execution_count": 2,
"id": "c45405ec",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "The X.TX matrix:[[4. 2. 2.]\n",
+ " [2. 2. 1.]\n",
+ " [2. 1. 2.]]\n",
+ "The invers of X.TX matrix:[[ 7.50000000e-01 -5.00000000e-01 -5.00000000e-01]\n",
+ " [-5.00000000e-01 1.00000000e+00 -2.22044605e-16]\n",
+ " [-5.00000000e-01 -4.44089210e-16 1.00000000e+00]]\n",
+ "The values of theta for the XOR gate:[ 5.00000000e-01 -1.66533454e-16 -4.44089210e-16]\n",
+ "The linear regression prediction for the XOR gate:[0.5 0.5 0.5 0.5]\n",
+ "The values of theta for the OR gate:[0.25 0.5 0.5 ]\n",
+ "The linear regression prediction for the OR gate:[0.25 0.75 0.75 1.25]\n",
+ "The values of theta for the AND gate:[-0.25 0.5 0.5 ]\n",
+ "The linear regression prediction for the AND gate:[-0.25 0.25 0.25 0.75]\n",
+ "Test set accuracy with Logistic Regression for OR gate: 0.75\n",
+ "Test set accuracy with Logistic Regression for XOR gate: 0.50\n",
+ "Test set accuracy with Logistic Regression for AND gate: 0.75\n"
+ ]
+ }
+ ],
"source": [
"%matplotlib inline\n",
"\n",
@@ -547,9 +538,7 @@
{
"cell_type": "markdown",
"id": "3a4e24b3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Not exactly impressive, but somewhat better."
]
@@ -557,9 +546,7 @@
{
"cell_type": "markdown",
"id": "a01ebeb0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Adding Neural Networks"
]
@@ -568,11 +555,16 @@
"cell_type": "code",
"execution_count": 3,
"id": "a1f650e1",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
- "outputs": [],
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Test set accuracy with Feed Forward Neural Network for XOR gate:1.0\n"
+ ]
+ }
+ ],
"source": [
"\n",
"# and now neural networks with Scikit-Learn and the XOR\n",
@@ -588,9 +580,7 @@
{
"cell_type": "markdown",
"id": "f7be1d28",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Mathematical model\n",
"\n",
@@ -600,9 +590,7 @@
{
"cell_type": "markdown",
"id": "3459f900",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"y = f\\left(\\sum_{i=1}^n w_ix_i + b_i\\right) = f(z),\n",
@@ -612,9 +600,7 @@
{
"cell_type": "markdown",
"id": "d8c3c549",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This function receives $x_i$ as inputs.\n",
"Here the activation $z=(\\sum_{i=1}^n w_ix_i+b_i)$. \n",
@@ -627,9 +613,7 @@
{
"cell_type": "markdown",
"id": "0c694b69",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Mathematical model\n",
"\n",
@@ -639,9 +623,7 @@
{
"cell_type": "markdown",
"id": "49b26503",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -656,9 +638,7 @@
{
"cell_type": "markdown",
"id": "a034397d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Here $b_i$ is the so-called bias which is normally needed in\n",
"case of zero activation weights or inputs. How to fix the biases and\n",
@@ -671,9 +651,7 @@
{
"cell_type": "markdown",
"id": "cb1bec7c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -689,9 +667,7 @@
{
"cell_type": "markdown",
"id": "1a9da712",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where we assume that all nodes in the same layer have identical\n",
"activation functions, hence the notation $f$. In general, we could assume in the more general case that different layers have different activation functions.\n",
@@ -701,9 +677,7 @@
{
"cell_type": "markdown",
"id": "af6c87ad",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -719,9 +693,7 @@
{
"cell_type": "markdown",
"id": "81bc149b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $N_l$ is the number of nodes in layer $l$. When the output of\n",
"all the nodes in the first hidden layer are computed, the values of\n",
@@ -732,9 +704,7 @@
{
"cell_type": "markdown",
"id": "f507c2d6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Mathematical model\n",
"\n",
@@ -744,9 +714,7 @@
{
"cell_type": "markdown",
"id": "b9251a8f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -762,9 +730,7 @@
{
"cell_type": "markdown",
"id": "833b1bfc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -780,9 +746,7 @@
{
"cell_type": "markdown",
"id": "daf16c0f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where we have substituted $y_k^1$ with the inputs $x_k$. Finally, the ANN output reads"
]
@@ -790,9 +754,7 @@
{
"cell_type": "markdown",
"id": "781c914b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -808,9 +770,7 @@
{
"cell_type": "markdown",
"id": "9c49f0a2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -827,9 +787,7 @@
{
"cell_type": "markdown",
"id": "76754fd8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Mathematical model\n",
"\n",
@@ -840,9 +798,7 @@
{
"cell_type": "markdown",
"id": "6fcdd054",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -858,9 +814,7 @@
{
"cell_type": "markdown",
"id": "565a89af",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which illustrates a basic property of MLPs: The only independent\n",
"variables are the input values $x_n$."
@@ -869,9 +823,7 @@
{
"cell_type": "markdown",
"id": "0e7a1ebb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Mathematical model\n",
"\n",
@@ -888,9 +840,7 @@
{
"cell_type": "markdown",
"id": "685ce363",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -906,9 +856,7 @@
{
"cell_type": "markdown",
"id": "f2e57f34",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where the parameters $c_i$ are weights and biases. By adjusting these\n",
"parameters, the activation functions can be shifted up and down or\n",
@@ -919,9 +867,7 @@
{
"cell_type": "markdown",
"id": "a648221d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"### Matrix-vector notation\n",
"\n",
@@ -939,9 +885,7 @@
{
"cell_type": "markdown",
"id": "1018cb53",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -972,9 +916,7 @@
{
"cell_type": "markdown",
"id": "d50fdc5a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"### Matrix-vector notation and activation\n",
"\n",
@@ -984,9 +926,7 @@
{
"cell_type": "markdown",
"id": "8a0e54b8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -1003,9 +943,7 @@
{
"cell_type": "markdown",
"id": "cdd4f871",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This is not just a convenient and compact notation, but also a useful\n",
"and intuitive way to think about MLPs: The output is calculated by a\n",
@@ -1017,9 +955,7 @@
{
"cell_type": "markdown",
"id": "8113f967",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"### Activation functions\n",
"\n",
@@ -1040,9 +976,7 @@
{
"cell_type": "markdown",
"id": "46c07259",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"### Activation functions, Logistic and Hyperbolic ones\n",
"\n",
@@ -1059,9 +993,7 @@
{
"cell_type": "markdown",
"id": "905c36c2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x) = \\frac{1}{1 + e^{-x}},\n",
@@ -1071,9 +1003,7 @@
{
"cell_type": "markdown",
"id": "d5178457",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and the *hyperbolic tangent* function"
]
@@ -1081,9 +1011,7 @@
{
"cell_type": "markdown",
"id": "fb82cff0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x) = \\tanh(x)\n",
@@ -1093,9 +1021,7 @@
{
"cell_type": "markdown",
"id": "ae8f128c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"### Relevance\n",
"\n",
@@ -1110,10 +1036,7 @@
"cell_type": "code",
"execution_count": 4,
"id": "52103b4a",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\"\"\"The sigmoid function (or the logistic curve) is a \n",
@@ -1192,9 +1115,7 @@
{
"cell_type": "markdown",
"id": "118d82f2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The multilayer perceptron (MLP)\n",
"\n",
@@ -1230,9 +1151,7 @@
{
"cell_type": "markdown",
"id": "62be696e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## From one to many layers, the universal approximation theorem\n",
"\n",
@@ -1260,9 +1179,7 @@
{
"cell_type": "markdown",
"id": "b172bc20",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Deriving the back propagation code for a multilayer perceptron model\n",
"\n",
@@ -1281,9 +1198,7 @@
{
"cell_type": "markdown",
"id": "926f66bc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"{\\cal C}(\\hat{W}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - t_i\\right)^2,\n",
@@ -1293,9 +1208,7 @@
{
"cell_type": "markdown",
"id": "0224162a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where the $t_i$s are our $n$ targets (the values we want to\n",
"reproduce), while the outputs of the network after having propagated\n",
@@ -1308,9 +1221,7 @@
{
"cell_type": "markdown",
"id": "19e36765",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Definitions\n",
"\n",
@@ -1325,9 +1236,7 @@
{
"cell_type": "markdown",
"id": "225a895c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_j^l = \\sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,\n",
@@ -1337,9 +1246,7 @@
{
"cell_type": "markdown",
"id": "ce115d2e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $b_k^l$ are the biases from layer $l$. Here $M_{l-1}$\n",
"represents the total number of nodes/neurons/units of layer $l-1$. The\n",
@@ -1350,9 +1257,7 @@
{
"cell_type": "markdown",
"id": "fa8127d3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\hat{z}^l = \\left(\\hat{W}^l\\right)^T\\hat{a}^{l-1}+\\hat{b}^l.\n",
@@ -1362,9 +1267,7 @@
{
"cell_type": "markdown",
"id": "611d0ae6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"With the activation values $\\hat{z}^l$ we can in turn define the\n",
"output of layer $l$ as $\\hat{a}^l = f(\\hat{z}^l)$ where $f$ is our\n",
@@ -1376,9 +1279,7 @@
{
"cell_type": "markdown",
"id": "60a383ee",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"a_j^l = f(z_j^l) = \\frac{1}{1+\\exp{-(z_j^l)}}.\n",
@@ -1388,9 +1289,7 @@
{
"cell_type": "markdown",
"id": "2a0a1798",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Derivatives and the chain rule\n",
"\n",
@@ -1400,9 +1299,7 @@
{
"cell_type": "markdown",
"id": "66ca578c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial z_j^l}{\\partial w_{ij}^l} = a_i^{l-1},\n",
@@ -1412,9 +1309,7 @@
{
"cell_type": "markdown",
"id": "99931f69",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1422,9 +1317,7 @@
{
"cell_type": "markdown",
"id": "13bce0c7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial z_j^l}{\\partial a_i^{l-1}} = w_{ji}^l.\n",
@@ -1434,9 +1327,7 @@
{
"cell_type": "markdown",
"id": "160cf1df",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"With our definition of the activation function we have that (note that this function depends only on $z_j^l$)"
]
@@ -1444,9 +1335,7 @@
{
"cell_type": "markdown",
"id": "82e6c32b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial a_j^l}{\\partial z_j^{l}} = a_j^l(1-a_j^l)=f(z_j^l)(1-f(z_j^l)).\n",
@@ -1456,9 +1345,7 @@
{
"cell_type": "markdown",
"id": "8c49c846",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Derivative of the cost function\n",
"\n",
@@ -1470,9 +1357,7 @@
{
"cell_type": "markdown",
"id": "917c676a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"{\\cal C}(\\hat{W^L}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - t_i\\right)^2=\\frac{1}{2}\\sum_{i=1}^n\\left(a_i^L - t_i\\right)^2,\n",
@@ -1482,9 +1367,7 @@
{
"cell_type": "markdown",
"id": "549cb9af",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The derivative of this function with respect to the weights is"
]
@@ -1492,9 +1375,7 @@
{
"cell_type": "markdown",
"id": "8b358bf8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}(\\hat{W^L})}{\\partial w_{jk}^L} = \\left(a_j^L - t_j\\right)\\frac{\\partial a_j^L}{\\partial w_{jk}^{L}},\n",
@@ -1504,9 +1385,7 @@
{
"cell_type": "markdown",
"id": "bfefb9a4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The last partial derivative can easily be computed and reads (by applying the chain rule)"
]
@@ -1514,9 +1393,7 @@
{
"cell_type": "markdown",
"id": "99e18ab1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial a_j^L}{\\partial w_{jk}^{L}} = \\frac{\\partial a_j^L}{\\partial z_{j}^{L}}\\frac{\\partial z_j^L}{\\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1},\n",
@@ -1526,9 +1403,7 @@
{
"cell_type": "markdown",
"id": "a00fdb28",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Bringing it together, first back propagation equation\n",
"\n",
@@ -1538,9 +1413,7 @@
{
"cell_type": "markdown",
"id": "dc30c543",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}(\\hat{W^L})}{\\partial w_{jk}^L} = \\left(a_j^L - t_j\\right)a_j^L(1-a_j^L)a_k^{L-1},\n",
@@ -1550,9 +1423,7 @@
{
"cell_type": "markdown",
"id": "599f8d24",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Defining"
]
@@ -1560,9 +1431,7 @@
{
"cell_type": "markdown",
"id": "0caf001e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L = a_j^L(1-a_j^L)\\left(a_j^L - t_j\\right) = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n",
@@ -1572,9 +1441,7 @@
{
"cell_type": "markdown",
"id": "ddbb3b09",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and using the Hadamard product of two vectors we can write this as"
]
@@ -1582,9 +1449,7 @@
{
"cell_type": "markdown",
"id": "13565ca9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\hat{\\delta}^L = f'(\\hat{z}^L)\\circ\\frac{\\partial {\\cal C}}{\\partial (\\hat{a}^L)}.\n",
@@ -1594,9 +1459,7 @@
{
"cell_type": "markdown",
"id": "5f880e7a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This is an important expression. The second term on the right handside\n",
"measures how fast the cost function is changing as a function of the $j$th\n",
@@ -1618,9 +1481,7 @@
{
"cell_type": "markdown",
"id": "f5b39e84",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}\n",
@@ -1630,9 +1491,7 @@
{
"cell_type": "markdown",
"id": "19a26ead",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"With the definition of $\\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely"
]
@@ -1640,9 +1499,7 @@
{
"cell_type": "markdown",
"id": "e36bbfdf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}(\\hat{W^L})}{\\partial w_{jk}^L} = \\delta_j^La_k^{L-1}.\n",
@@ -1652,9 +1509,7 @@
{
"cell_type": "markdown",
"id": "645aea25",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Derivatives in terms of $z_j^L$\n",
"\n",
@@ -1664,9 +1519,7 @@
{
"cell_type": "markdown",
"id": "a9081be7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L =\\frac{\\partial {\\cal C}}{\\partial z_j^L}= \\frac{\\partial {\\cal C}}{\\partial a_j^L}\\frac{\\partial a_j^L}{\\partial z_j^L},\n",
@@ -1676,9 +1529,7 @@
{
"cell_type": "markdown",
"id": "23dec8ab",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely"
]
@@ -1686,9 +1537,7 @@
{
"cell_type": "markdown",
"id": "702b4a0c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L}\\frac{\\partial b_j^L}{\\partial z_j^L}=\\frac{\\partial {\\cal C}}{\\partial b_j^L},\n",
@@ -1698,9 +1547,7 @@
{
"cell_type": "markdown",
"id": "a2e76cc4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"That is, the error $\\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias."
]
@@ -1708,9 +1555,7 @@
{
"cell_type": "markdown",
"id": "d80aa949",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Bringing it together\n",
"\n",
@@ -1722,9 +1567,7 @@
{
"cell_type": "markdown",
"id": "168bb344",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -1740,9 +1583,7 @@
{
"cell_type": "markdown",
"id": "c2c9b8ee",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1750,9 +1591,7 @@
{
"cell_type": "markdown",
"id": "394b40ec",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -1768,9 +1607,7 @@
{
"cell_type": "markdown",
"id": "975b96e1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1778,9 +1615,7 @@
{
"cell_type": "markdown",
"id": "9171b598",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -1796,9 +1631,7 @@
{
"cell_type": "markdown",
"id": "d69ea77c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"An interesting consequence of the above equations is that when the\n",
"activation $a_k^{L-1}$ is small, the gradient term, that is the\n",
@@ -1823,9 +1656,7 @@
{
"cell_type": "markdown",
"id": "2d94ea5e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final back propagating equation\n",
"\n",
@@ -1835,9 +1666,7 @@
{
"cell_type": "markdown",
"id": "3171e9da",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l =\\frac{\\partial {\\cal C}}{\\partial z_j^l}.\n",
@@ -1847,9 +1676,7 @@
{
"cell_type": "markdown",
"id": "4572f975",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We want to express this in terms of the equations for layer $l+1$. Using the chain rule and summing over all $k$ entries we have"
]
@@ -1857,9 +1684,7 @@
{
"cell_type": "markdown",
"id": "eaa6ba5f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l =\\sum_k \\frac{\\partial {\\cal C}}{\\partial z_k^{l+1}}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}}=\\sum_k \\delta_k^{l+1}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}},\n",
@@ -1869,9 +1694,7 @@
{
"cell_type": "markdown",
"id": "36c6d661",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and recalling that"
]
@@ -1879,9 +1702,7 @@
{
"cell_type": "markdown",
"id": "841b16f8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_j^{l+1} = \\sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},\n",
@@ -1891,9 +1712,7 @@
{
"cell_type": "markdown",
"id": "724449b6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $M_l$ being the number of nodes in layer $l$, we obtain"
]
@@ -1901,9 +1720,7 @@
{
"cell_type": "markdown",
"id": "b7e095ea",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l =\\sum_k \\delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l),\n",
@@ -1913,9 +1730,7 @@
{
"cell_type": "markdown",
"id": "b21011a1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This is our final equation.\n",
"\n",
@@ -1925,9 +1740,7 @@
{
"cell_type": "markdown",
"id": "b8166607",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the Back propagation algorithm\n",
"\n",
@@ -1948,9 +1761,7 @@
{
"cell_type": "markdown",
"id": "988ad4d1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n",
@@ -1960,9 +1771,7 @@
{
"cell_type": "markdown",
"id": "f303a183",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as"
]
@@ -1970,9 +1779,7 @@
{
"cell_type": "markdown",
"id": "8671fa8a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).\n",
@@ -1982,9 +1789,7 @@
{
"cell_type": "markdown",
"id": "3f645bd9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules"
]
@@ -1992,9 +1797,7 @@
{
"cell_type": "markdown",
"id": "ebc0d0e1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n",
@@ -2004,9 +1807,7 @@
{
"cell_type": "markdown",
"id": "e0db6344",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
@@ -2016,9 +1817,7 @@
{
"cell_type": "markdown",
"id": "967460f6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n",
"Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training."
@@ -2027,9 +1826,7 @@
{
"cell_type": "markdown",
"id": "a7357dd6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the Back propagation algorithm\n",
"\n",
@@ -2050,9 +1847,7 @@
{
"cell_type": "markdown",
"id": "fc37ea55",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n",
@@ -2062,9 +1857,7 @@
{
"cell_type": "markdown",
"id": "6be0fe38",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as"
]
@@ -2072,9 +1865,7 @@
{
"cell_type": "markdown",
"id": "fb56ae32",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).\n",
@@ -2084,9 +1875,7 @@
{
"cell_type": "markdown",
"id": "97e480ea",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules"
]
@@ -2094,9 +1883,7 @@
{
"cell_type": "markdown",
"id": "302971ad",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n",
@@ -2106,9 +1893,7 @@
{
"cell_type": "markdown",
"id": "8ddb6465",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
@@ -2118,9 +1903,7 @@
{
"cell_type": "markdown",
"id": "1b82b6c7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n",
"Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training."
@@ -2129,9 +1912,7 @@
{
"cell_type": "markdown",
"id": "1ec58322",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up a Multi-layer perceptron model for classification\n",
"\n",
@@ -2157,9 +1938,7 @@
{
"cell_type": "markdown",
"id": "0280ce72",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp{(- \\boldsymbol{x}})} ,\n",
@@ -2169,9 +1948,7 @@
{
"cell_type": "markdown",
"id": "8ad11f29",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -2179,9 +1956,7 @@
{
"cell_type": "markdown",
"id": "00afc6f4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"P(y = 1 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = 1 - P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) ,\n",
@@ -2191,9 +1966,7 @@
{
"cell_type": "markdown",
"id": "dc35157b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $y \\in \\{0, 1\\}$ and $\\boldsymbol{\\theta}$ represents the weights and biases\n",
"of our network."
@@ -2202,9 +1975,7 @@
{
"cell_type": "markdown",
"id": "92fff047",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Defining the cost function\n",
"\n",
@@ -2214,9 +1985,7 @@
{
"cell_type": "markdown",
"id": "76faa893",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\theta}) = - \\ln P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = - \\sum_{i=1}^n\n",
@@ -2227,9 +1996,7 @@
{
"cell_type": "markdown",
"id": "1fc9a336",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This last equality means that we can interpret our *cost* function as a sum over the *loss* function\n",
"for each point in the dataset $\\mathcal{L}_i(\\boldsymbol{\\theta})$. \n",
@@ -2252,9 +2019,7 @@
{
"cell_type": "markdown",
"id": "d3da2b42",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"P(y_{ic} = 1 \\mid \\boldsymbol{x}_i, \\boldsymbol{\\theta}) = \\frac{\\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_c)}}\n",
@@ -2265,9 +2030,7 @@
{
"cell_type": "markdown",
"id": "81c3e83e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which reduces to the logistic function in the binary case. \n",
"The likelihood of this $C$-class classifier\n",
@@ -2277,9 +2040,7 @@
{
"cell_type": "markdown",
"id": "482a90f4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = \\prod_{i=1}^n \\prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} .\n",
@@ -2289,9 +2050,7 @@
{
"cell_type": "markdown",
"id": "aad7903f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Again we take the negative log-likelihood to define our cost function:"
]
@@ -2299,9 +2058,7 @@
{
"cell_type": "markdown",
"id": "7286605e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\theta}) = - \\log{P(\\mathcal{D} \\mid \\boldsymbol{\\theta})}.\n",
@@ -2311,9 +2068,7 @@
{
"cell_type": "markdown",
"id": "d62a965a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"See the logistic regression lectures for a full definition of the cost function.\n",
"\n",
@@ -2323,9 +2078,7 @@
{
"cell_type": "markdown",
"id": "ef5e5820",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Example: binary classification problem\n",
"\n",
@@ -2335,9 +2088,7 @@
{
"cell_type": "markdown",
"id": "5ff39e0d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = - \\sum_{i=1}^n \\left(y_i\\log{p(y_i \\vert x_i,\\boldsymbol{\\beta})}+(1-y_i)\\log{1-p(y_i \\vert x_i,\\boldsymbol{\\beta})}\\right),\n",
@@ -2347,9 +2098,7 @@
{
"cell_type": "markdown",
"id": "40227130",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where we had defined the logistic (sigmoid) function"
]
@@ -2357,9 +2106,7 @@
{
"cell_type": "markdown",
"id": "f75ef3d0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(y_i =1\\vert x_i,\\boldsymbol{\\beta})=\\frac{\\exp{(\\beta_0+\\beta_1 x_i)}}{1+\\exp{(\\beta_0+\\beta_1 x_i)}},\n",
@@ -2369,9 +2116,7 @@
{
"cell_type": "markdown",
"id": "2a366112",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -2379,9 +2124,7 @@
{
"cell_type": "markdown",
"id": "2d863c22",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(y_i =0\\vert x_i,\\boldsymbol{\\beta})=1-p(y_i =1\\vert x_i,\\boldsymbol{\\beta}).\n",
@@ -2391,9 +2134,7 @@
{
"cell_type": "markdown",
"id": "9c0dfa23",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The parameters $\\boldsymbol{\\beta}$ were defined using a minimization method like gradient descent or Newton-Raphson's method. \n",
"\n",
@@ -2404,9 +2145,7 @@
{
"cell_type": "markdown",
"id": "14971d06",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"a_i^l = y_i = \\frac{\\exp{(z_i^l)}}{1+\\exp{(z_i^l)}},\n",
@@ -2416,9 +2155,7 @@
{
"cell_type": "markdown",
"id": "67c5f615",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with"
]
@@ -2426,9 +2163,7 @@
{
"cell_type": "markdown",
"id": "4f0c1881",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_i^l = \\sum_{j}w_{ij}^l a_j^{l-1}+b_i^l,\n",
@@ -2438,9 +2173,7 @@
{
"cell_type": "markdown",
"id": "1cc31fb6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where the superscript $l-1$ indicates that these are the outputs from layer $l-1$.\n",
"Our cost function at the final layer $l=L$ is now"
@@ -2449,9 +2182,7 @@
{
"cell_type": "markdown",
"id": "ac81435e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{W}) = - \\sum_{i=1}^n \\left(t_i\\log{a_i^L}+(1-t_i)\\log{(1-a_i^L)}\\right),\n",
@@ -2461,9 +2192,7 @@
{
"cell_type": "markdown",
"id": "de35fd7e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where we have defined the targets $t_i$. The derivatives of the cost function with respect to the output $a_i^L$ are then easily calculated and we get"
]
@@ -2471,9 +2200,7 @@
{
"cell_type": "markdown",
"id": "7e4c1efd",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{W})}{\\partial a_i^L} = \\frac{a_i^L-t_i}{a_i^L(1-a_i^L)}.\n",
@@ -2483,9 +2210,7 @@
{
"cell_type": "markdown",
"id": "b4418588",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"In case we use another activation function than the logistic one, we need to evaluate other derivatives."
]
@@ -2493,9 +2218,7 @@
{
"cell_type": "markdown",
"id": "771b710c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The Softmax function\n",
"In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need"
@@ -2504,9 +2227,7 @@
{
"cell_type": "markdown",
"id": "1e1f7cf2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f(z_i^l)}{\\partial w_{jk}^l} =\n",
@@ -2517,9 +2238,7 @@
{
"cell_type": "markdown",
"id": "d2de3ace",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"For the Softmax function we have"
]
@@ -2527,9 +2246,7 @@
{
"cell_type": "markdown",
"id": "c90dd2a5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(z_i^l) = \\frac{\\exp{(z_i^l)}}{\\sum_{m=1}^K\\exp{(z_m^l)}}.\n",
@@ -2539,9 +2256,7 @@
{
"cell_type": "markdown",
"id": "c2f8e81e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Its derivative with respect to $z_j^l$ gives"
]
@@ -2549,9 +2264,7 @@
{
"cell_type": "markdown",
"id": "a96a56dc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f(z_i^l)}{\\partial z_j^l}= f(z_i^l)\\left(\\delta_{ij}-f(z_j^l)\\right),\n",
@@ -2561,9 +2274,7 @@
{
"cell_type": "markdown",
"id": "6ff2dc62",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which in case of the simply binary model reduces to having $i=j$."
]
@@ -2571,9 +2282,7 @@
{
"cell_type": "markdown",
"id": "821e9ed7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Developing a code for doing neural networks with back propagation\n",
"\n",
@@ -2595,9 +2304,7 @@
{
"cell_type": "markdown",
"id": "100d59d3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Collect and pre-process data\n",
"\n",
@@ -2645,10 +2352,7 @@
"cell_type": "code",
"execution_count": 5,
"id": "80a7da13",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -2698,9 +2402,7 @@
{
"cell_type": "markdown",
"id": "10dae5de",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Train and test datasets\n",
"\n",
@@ -2719,10 +2421,7 @@
"cell_type": "code",
"execution_count": 6,
"id": "4ea5079b",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
@@ -2757,9 +2456,7 @@
{
"cell_type": "markdown",
"id": "6998ce20",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Define model and architecture\n",
"\n",
@@ -2801,9 +2498,7 @@
{
"cell_type": "markdown",
"id": "4820ddf9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Layers\n",
"\n",
@@ -2841,9 +2536,7 @@
{
"cell_type": "markdown",
"id": "089eda51",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Weights and biases\n",
"\n",
@@ -2862,10 +2555,7 @@
"cell_type": "code",
"execution_count": 7,
"id": "b279bfa7",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# building our neural network\n",
@@ -2888,9 +2578,7 @@
{
"cell_type": "markdown",
"id": "ea232201",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Feed-forward pass\n",
"\n",
@@ -2916,9 +2604,7 @@
{
"cell_type": "markdown",
"id": "c4128578",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Matrix multiplications\n",
"\n",
@@ -2953,10 +2639,7 @@
"cell_type": "code",
"execution_count": 8,
"id": "c1386e6d",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# setup the feed-forward pass, subscript h = hidden layer\n",
@@ -2999,9 +2682,7 @@
{
"cell_type": "markdown",
"id": "0e3b21fb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Choose cost function and optimizer\n",
"\n",
@@ -3030,9 +2711,7 @@
{
"cell_type": "markdown",
"id": "b9aceefb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Optimizing the cost function\n",
"\n",
@@ -3068,9 +2747,7 @@
{
"cell_type": "markdown",
"id": "adf1ca53",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Regularization\n",
"\n",
@@ -3102,9 +2779,7 @@
{
"cell_type": "markdown",
"id": "55c9468d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Matrix multiplication\n",
"\n",
@@ -3143,10 +2818,7 @@
"cell_type": "code",
"execution_count": 9,
"id": "d7369c6d",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# to categorical turns our integer vector into a onehot representation\n",
@@ -3222,9 +2894,7 @@
{
"cell_type": "markdown",
"id": "9c0c2f1d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Improving performance\n",
"\n",
@@ -3243,9 +2913,7 @@
{
"cell_type": "markdown",
"id": "73961632",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Full object-oriented implementation\n",
"\n",
@@ -3257,10 +2925,7 @@
"cell_type": "code",
"execution_count": 10,
"id": "4405e00e",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"class NeuralNetwork:\n",
@@ -3367,9 +3032,7 @@
{
"cell_type": "markdown",
"id": "518d89b2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Evaluate model performance on test data\n",
"\n",
@@ -3386,10 +3049,7 @@
"cell_type": "code",
"execution_count": 11,
"id": "afafc202",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"epochs = 100\n",
@@ -3413,9 +3073,7 @@
{
"cell_type": "markdown",
"id": "6c49cfa6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Adjust hyperparameters\n",
"\n",
@@ -3427,10 +3085,7 @@
"cell_type": "code",
"execution_count": 12,
"id": "48281e23",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"eta_vals = np.logspace(-5, 1, 7)\n",
@@ -3458,9 +3113,7 @@
{
"cell_type": "markdown",
"id": "bae5fc53",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Visualization"
]
@@ -3469,10 +3122,7 @@
"cell_type": "code",
"execution_count": 13,
"id": "6ee4615a",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# visual representation of grid search\n",
@@ -3513,9 +3163,7 @@
{
"cell_type": "markdown",
"id": "379e36d7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## scikit-learn implementation\n",
"\n",
@@ -3536,10 +3184,7 @@
"cell_type": "code",
"execution_count": 14,
"id": "322af1dc",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"from sklearn.neural_network import MLPClassifier\n",
@@ -3563,9 +3208,7 @@
{
"cell_type": "markdown",
"id": "475c82ca",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Visualization"
]
@@ -3574,10 +3217,7 @@
"cell_type": "code",
"execution_count": 15,
"id": "076354f9",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# optional\n",
@@ -3617,7 +3257,25 @@
]
}
],
- "metadata": {},
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.14"
+ }
+ },
"nbformat": 4,
"nbformat_minor": 5
}