diff --git a/doc/pub/week41/ipynb/week41.ipynb b/doc/pub/week41/ipynb/week41.ipynb
index 8d2109293..5e4352c71 100644
--- a/doc/pub/week41/ipynb/week41.ipynb
+++ b/doc/pub/week41/ipynb/week41.ipynb
@@ -3,9 +3,7 @@
{
"cell_type": "markdown",
"id": "5bf96c8a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
@@ -15,9 +13,7 @@
{
"cell_type": "markdown",
"id": "e3d91be4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"# Week 41 Neural networks and constructing a neural network code\n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo\n",
@@ -28,9 +24,7 @@
{
"cell_type": "markdown",
"id": "6ba7a03a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Plan for week 41, October 7-11"
]
@@ -38,9 +32,7 @@
{
"cell_type": "markdown",
"id": "e8ee6c3d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Material for the lecture on Monday October 7, 2024\n",
"1. Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.\n",
@@ -73,9 +65,7 @@
{
"cell_type": "markdown",
"id": "f1d45fb2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Material for the active learning sessions on Tuesday and Wednesday\n",
"* Exercise on writing your own stochastic gradient and gradient descent codes. This exercise continues next week with studies of automatic differentiation\n",
@@ -90,9 +80,7 @@
{
"cell_type": "markdown",
"id": "d4e77966",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Lecture Monday October 7"
]
@@ -100,9 +88,7 @@
{
"cell_type": "markdown",
"id": "e53c7266",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Introduction to Neural networks\n",
"\n",
@@ -118,9 +104,7 @@
{
"cell_type": "markdown",
"id": "bc8b8894",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Artificial neurons\n",
"\n",
@@ -142,9 +126,7 @@
{
"cell_type": "markdown",
"id": "7822604f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -160,9 +142,7 @@
{
"cell_type": "markdown",
"id": "e1b48997",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Here, the output $y$ of the neuron is the value of its activation function, which have as input\n",
"a weighted sum of signals $x_i, \\dots ,x_n$ received by $n$ other neurons.\n",
@@ -200,9 +180,7 @@
{
"cell_type": "markdown",
"id": "17881d26",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Neural network types\n",
"\n",
@@ -229,9 +207,7 @@
{
"cell_type": "markdown",
"id": "64134ea5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Feed-forward neural networks\n",
"\n",
@@ -250,9 +226,7 @@
{
"cell_type": "markdown",
"id": "d1b8101c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Convolutional Neural Network\n",
"\n",
@@ -279,9 +253,7 @@
{
"cell_type": "markdown",
"id": "ee76d630",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Recurrent neural networks\n",
"\n",
@@ -300,9 +272,7 @@
{
"cell_type": "markdown",
"id": "77292572",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Other types of networks\n",
"\n",
@@ -321,9 +291,7 @@
{
"cell_type": "markdown",
"id": "31229c23",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Multilayer perceptrons\n",
"\n",
@@ -338,9 +306,7 @@
{
"cell_type": "markdown",
"id": "8ef64a1c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Why multilayer perceptrons?\n",
"\n",
@@ -359,9 +325,7 @@
{
"cell_type": "markdown",
"id": "8a05aae8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Illustration of a single perceptron model and a multi-perceptron model\n",
"\n",
@@ -375,9 +339,7 @@
{
"cell_type": "markdown",
"id": "53553bc4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Examples of XOR, OR and AND gates\n",
"\n",
@@ -392,10 +354,7 @@
"cell_type": "code",
"execution_count": 1,
"id": "bd160078",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
@@ -433,9 +392,7 @@
{
"cell_type": "markdown",
"id": "8dc03535",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"What is happening here?"
]
@@ -443,9 +400,7 @@
{
"cell_type": "markdown",
"id": "0f612b2a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Does Logistic Regression do a better Job?"
]
@@ -454,10 +409,7 @@
"cell_type": "code",
"execution_count": 2,
"id": "b2c26240",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -516,9 +468,7 @@
{
"cell_type": "markdown",
"id": "f5dc459e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Not exactly impressive, but somewhat better."
]
@@ -526,9 +476,7 @@
{
"cell_type": "markdown",
"id": "b2ad997b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Adding Neural Networks"
]
@@ -537,10 +485,7 @@
"cell_type": "code",
"execution_count": 3,
"id": "df5c9c62",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\n",
@@ -557,9 +502,7 @@
{
"cell_type": "markdown",
"id": "40a0930e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Mathematics of deep learning\n",
"\n",
@@ -573,9 +516,7 @@
{
"cell_type": "markdown",
"id": "decd7409",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Reminder on books with hands-on material and codes\n",
"* [Sebastian Rashcka et al, Machine learning with Sickit-Learn and PyTorch](https://sebastianraschka.com/blog/2022/ml-pytorch-book.html)"
@@ -584,9 +525,7 @@
{
"cell_type": "markdown",
"id": "e2a4d61c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Reading recommendations\n",
"\n",
@@ -598,9 +537,7 @@
{
"cell_type": "markdown",
"id": "16723d55",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Mathematics of deep learning and neural networks\n",
"\n",
@@ -617,9 +554,7 @@
{
"cell_type": "markdown",
"id": "902a802b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Basics of an NN\n",
"\n",
@@ -641,9 +576,7 @@
{
"cell_type": "markdown",
"id": "fbd06338",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Overarching view of a neural network\n",
"\n",
@@ -668,9 +601,7 @@
{
"cell_type": "markdown",
"id": "0e9e449b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The optimization problem\n",
"\n",
@@ -685,9 +616,7 @@
{
"cell_type": "markdown",
"id": "b06ad67d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C(\\boldsymbol{\\Theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
@@ -697,9 +626,7 @@
{
"cell_type": "markdown",
"id": "aa068f3c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This function represents one of many possible ways to define\n",
"the so-called cost function. Note that here we have assumed a linear dependence in terms of the paramters $\\boldsymbol{\\Theta}$. This is in general not the case."
@@ -708,9 +635,7 @@
{
"cell_type": "markdown",
"id": "2977103a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Parameters of neural networks\n",
"For neural networks the parameters\n",
@@ -726,9 +651,7 @@
{
"cell_type": "markdown",
"id": "3cd55846",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Other ingredients of a neural network\n",
"\n",
@@ -755,9 +678,7 @@
{
"cell_type": "markdown",
"id": "66def2af",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Other parameters\n",
"\n",
@@ -769,9 +690,7 @@
{
"cell_type": "markdown",
"id": "1dfc75e0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Universal approximation theorem\n",
"\n",
@@ -785,9 +704,7 @@
{
"cell_type": "markdown",
"id": "db4ec583",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\sigma(z) = \\left\\{\\begin{array}{cc} 1 & z\\rightarrow \\infty\\\\ 0 & z \\rightarrow -\\infty \\end{array}\\right.\n",
@@ -797,9 +714,7 @@
{
"cell_type": "markdown",
"id": "e6b6f0e5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Given a continuous and deterministic function $F(\\boldsymbol{x})$ on the unit\n",
"cube in $d$-dimensions $F\\in [0,1]^d$, $x\\in [0,1]^d$ and a parameter\n",
@@ -811,9 +726,7 @@
{
"cell_type": "markdown",
"id": "82c4e644",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\vert F(\\boldsymbol{x})-f(\\boldsymbol{x};\\boldsymbol{\\Theta})\\vert < \\epsilon \\hspace{0.1cm} \\forall \\boldsymbol{x}\\in[0,1]^d.\n",
@@ -823,9 +736,7 @@
{
"cell_type": "markdown",
"id": "4d8b622d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Some parallels from real analysis\n",
"\n",
@@ -839,9 +750,7 @@
{
"cell_type": "markdown",
"id": "8d93823d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The approximation theorem in words\n",
"\n",
@@ -855,9 +764,7 @@
{
"cell_type": "markdown",
"id": "087fd46d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathbb{E}[\\vert F(\\boldsymbol{x})\\vert^2] =\\int_{\\boldsymbol{x}\\in D} \\vert F(\\boldsymbol{x})\\vert^2p(\\boldsymbol{x})d\\boldsymbol{x} < \\infty.\n",
@@ -867,9 +774,7 @@
{
"cell_type": "markdown",
"id": "7547efaa",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Then we have"
]
@@ -877,9 +782,7 @@
{
"cell_type": "markdown",
"id": "65071f00",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathbb{E}[\\vert F(\\boldsymbol{x})-f(\\boldsymbol{x};\\boldsymbol{\\Theta})\\vert^2] =\\int_{\\boldsymbol{x}\\in D} \\vert F(\\boldsymbol{x})-f(\\boldsymbol{x};\\boldsymbol{\\Theta})\\vert^2p(\\boldsymbol{x})d\\boldsymbol{x} < \\epsilon.\n",
@@ -889,9 +792,7 @@
{
"cell_type": "markdown",
"id": "7a8ecd20",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More on the general approximation theorem\n",
"\n",
@@ -907,9 +808,7 @@
{
"cell_type": "markdown",
"id": "41ed53e6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Class of functions we can approximate\n",
"\n",
@@ -920,9 +819,7 @@
{
"cell_type": "markdown",
"id": "2b83edda",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the equations for a neural network\n",
"\n",
@@ -937,9 +834,7 @@
{
"cell_type": "markdown",
"id": "d27eb1c2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"{\\cal C}(\\boldsymbol{\\Theta}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2,\n",
@@ -949,9 +844,7 @@
{
"cell_type": "markdown",
"id": "f83f5909",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where the $y_i$s are our $n$ targets (the values we want to\n",
"reproduce), while the outputs of the network after having propagated\n",
@@ -961,9 +854,7 @@
{
"cell_type": "markdown",
"id": "693f1f94",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Layout of a neural network with three hidden layers\n",
"\n",
@@ -977,9 +868,7 @@
{
"cell_type": "markdown",
"id": "c4473918",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Definitions\n",
"\n",
@@ -994,9 +883,7 @@
{
"cell_type": "markdown",
"id": "80c49202",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_j^l = \\sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,\n",
@@ -1006,9 +893,7 @@
{
"cell_type": "markdown",
"id": "b95ea794",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $b_k^l$ are the biases from layer $l$. Here $M_{l-1}$\n",
"represents the total number of nodes/neurons/units of layer $l-1$. The\n",
@@ -1019,9 +904,7 @@
{
"cell_type": "markdown",
"id": "7468be96",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\hat{z}^l = \\left(\\hat{W}^l\\right)^T\\hat{a}^{l-1}+\\hat{b}^l.\n",
@@ -1031,9 +914,7 @@
{
"cell_type": "markdown",
"id": "82d9ddc5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Inputs to the activation function\n",
"\n",
@@ -1047,9 +928,7 @@
{
"cell_type": "markdown",
"id": "611a8aaf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"a_j^l = \\sigma(z_j^l) = \\frac{1}{1+\\exp{-(z_j^l)}}.\n",
@@ -1059,9 +938,7 @@
{
"cell_type": "markdown",
"id": "adfdba0b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Derivatives and the chain rule\n",
"\n",
@@ -1071,9 +948,7 @@
{
"cell_type": "markdown",
"id": "6b23deb7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial z_j^l}{\\partial w_{ij}^l} = a_i^{l-1},\n",
@@ -1083,9 +958,7 @@
{
"cell_type": "markdown",
"id": "3dbf8b63",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1093,9 +966,7 @@
{
"cell_type": "markdown",
"id": "b3228dee",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial z_j^l}{\\partial a_i^{l-1}} = w_{ji}^l.\n",
@@ -1105,9 +976,7 @@
{
"cell_type": "markdown",
"id": "52be372b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"With our definition of the activation function we have that (note that this function depends only on $z_j^l$)"
]
@@ -1115,9 +984,7 @@
{
"cell_type": "markdown",
"id": "8e1cfc96",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial a_j^l}{\\partial z_j^{l}} = a_j^l(1-a_j^l)=\\sigma(z_j^l)(1-\\sigma(z_j^l)).\n",
@@ -1127,9 +994,7 @@
{
"cell_type": "markdown",
"id": "79b3ab6b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Derivative of the cost function\n",
"\n",
@@ -1141,9 +1006,7 @@
{
"cell_type": "markdown",
"id": "4541109b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"{\\cal C}(\\boldsymbol{\\Theta}^L) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2=\\frac{1}{2}\\sum_{i=1}^n\\left(a_i^L - y_i\\right)^2,\n",
@@ -1153,9 +1016,7 @@
{
"cell_type": "markdown",
"id": "ada15490",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The derivative of this function with respect to the weights is"
]
@@ -1163,9 +1024,7 @@
{
"cell_type": "markdown",
"id": "ef1846b1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}(\\boldsymbol{\\Theta}^L)}{\\partial w_{jk}^L} = \\left(a_j^L - y_j\\right)\\frac{\\partial a_j^L}{\\partial w_{jk}^{L}},\n",
@@ -1175,9 +1034,7 @@
{
"cell_type": "markdown",
"id": "2bcae885",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The last partial derivative can easily be computed and reads (by applying the chain rule)"
]
@@ -1185,9 +1042,7 @@
{
"cell_type": "markdown",
"id": "14438533",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial a_j^L}{\\partial w_{jk}^{L}} = \\frac{\\partial a_j^L}{\\partial z_{j}^{L}}\\frac{\\partial z_j^L}{\\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1}.\n",
@@ -1197,9 +1052,7 @@
{
"cell_type": "markdown",
"id": "1ea04dc6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Simpler examples first, and automatic differentiation\n",
"\n",
@@ -1212,9 +1065,7 @@
{
"cell_type": "markdown",
"id": "2c1d3d23",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Reminder on the chain rule and gradients\n",
"\n",
@@ -1224,9 +1075,7 @@
{
"cell_type": "markdown",
"id": "0a784486",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{df}{dt} = \\begin{bmatrix}\\frac{\\partial f}{\\partial x} & \\frac{\\partial f}{\\partial y} \\end{bmatrix} \\begin{bmatrix}\\frac{\\partial x}{\\partial t} \\\\ \\frac{\\partial y}{\\partial t} \\end{bmatrix}=\\frac{\\partial f}{\\partial x} \\frac{\\partial x}{\\partial t} +\\frac{\\partial f}{\\partial y} \\frac{\\partial y}{\\partial t}.\n",
@@ -1236,9 +1085,7 @@
{
"cell_type": "markdown",
"id": "91a6fac4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Multivariable functions\n",
"\n",
@@ -1248,9 +1095,7 @@
{
"cell_type": "markdown",
"id": "64245345",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial s}=\\frac{\\partial f}{\\partial x}\\frac{\\partial x}{\\partial s}+\\frac{\\partial f}{\\partial y}\\frac{\\partial y}{\\partial s},\n",
@@ -1260,9 +1105,7 @@
{
"cell_type": "markdown",
"id": "8f42f6ef",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1270,9 +1113,7 @@
{
"cell_type": "markdown",
"id": "23f3f62a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial t}=\\frac{\\partial f}{\\partial x}\\frac{\\partial x}{\\partial t}+\\frac{\\partial f}{\\partial y}\\frac{\\partial y}{\\partial t}.\n",
@@ -1282,9 +1123,7 @@
{
"cell_type": "markdown",
"id": "31db67ce",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"the gradient of $f$ with respect to $t$ and $s$ (without the explicit unit vector components)"
]
@@ -1292,9 +1131,7 @@
{
"cell_type": "markdown",
"id": "54f91226",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{df}{d(s,t)} = \\begin{bmatrix}\\frac{\\partial f}{\\partial x} & \\frac{\\partial f}{\\partial y} \\end{bmatrix} \\begin{bmatrix}\\frac{\\partial x}{\\partial s} &\\frac{\\partial x}{\\partial t} \\\\ \\frac{\\partial y}{\\partial s} & \\frac{\\partial y}{\\partial t} \\end{bmatrix}.\n",
@@ -1304,9 +1141,7 @@
{
"cell_type": "markdown",
"id": "551bf4e9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Automatic differentiation through examples\n",
"\n",
@@ -1324,9 +1159,7 @@
{
"cell_type": "markdown",
"id": "1d4117b4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Simple example\n",
"\n",
@@ -1336,9 +1169,7 @@
{
"cell_type": "markdown",
"id": "402669ea",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x) =\\exp{x^2},\n",
@@ -1348,9 +1179,7 @@
{
"cell_type": "markdown",
"id": "ec119701",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with derivative"
]
@@ -1358,9 +1187,7 @@
{
"cell_type": "markdown",
"id": "bcb30e6f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f'(x) =2x\\exp{x^2}.\n",
@@ -1370,9 +1197,7 @@
{
"cell_type": "markdown",
"id": "2a7acb6e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We can use SymPy to extract the pertinent lines of Python code through the following simple example"
]
@@ -1381,10 +1206,7 @@
"cell_type": "code",
"execution_count": 4,
"id": "044e79d9",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"from __future__ import division\n",
@@ -1400,9 +1222,7 @@
{
"cell_type": "markdown",
"id": "1c3cb722",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Smarter way of evaluating the above function\n",
"If we study this function, we note that we can reduce the number of operations by introducing an intermediate variable"
@@ -1411,9 +1231,7 @@
{
"cell_type": "markdown",
"id": "808a017a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"a = x^2,\n",
@@ -1423,9 +1241,7 @@
{
"cell_type": "markdown",
"id": "2330f35c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"leading to"
]
@@ -1433,9 +1249,7 @@
{
"cell_type": "markdown",
"id": "c0b17b83",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x) = f(a(x)) = b= \\exp{a}.\n",
@@ -1445,9 +1259,7 @@
{
"cell_type": "markdown",
"id": "9129586e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We now assume that all operations can be counted in terms of equal\n",
"floating point operations. This means that in order to calculate\n",
@@ -1458,9 +1270,7 @@
{
"cell_type": "markdown",
"id": "83209806",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Reducing the number of operations\n",
"\n",
@@ -1470,9 +1280,7 @@
{
"cell_type": "markdown",
"id": "1e04f023",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f'(x) = 2xb,\n",
@@ -1482,9 +1290,7 @@
{
"cell_type": "markdown",
"id": "4015de5c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which reduces the number of operations from four in the orginal\n",
"expression to two. This means that if we need to compute $f(x)$ and\n",
@@ -1499,9 +1305,7 @@
{
"cell_type": "markdown",
"id": "cfd489e0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Chain rule, forward and reverse modes\n",
"\n",
@@ -1511,9 +1315,7 @@
{
"cell_type": "markdown",
"id": "abcfa23a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x) = f(a(x)) = b= \\exp{a},\n",
@@ -1523,9 +1325,7 @@
{
"cell_type": "markdown",
"id": "67bcaa0e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $a=x^2$. We can decompose the derivative of $f$ with respect to $x$ as"
]
@@ -1533,9 +1333,7 @@
{
"cell_type": "markdown",
"id": "d4e12796",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{df}{dx}=\\frac{df}{db}\\frac{db}{da}\\frac{da}{dx}.\n",
@@ -1545,9 +1343,7 @@
{
"cell_type": "markdown",
"id": "e74eed87",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We note that since $b=f(x)$ that"
]
@@ -1555,9 +1351,7 @@
{
"cell_type": "markdown",
"id": "5b626985",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{df}{db}=1,\n",
@@ -1567,9 +1361,7 @@
{
"cell_type": "markdown",
"id": "4b91e7a2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"leading to"
]
@@ -1577,9 +1369,7 @@
{
"cell_type": "markdown",
"id": "6da09721",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{df}{dx}=\\frac{db}{da}\\frac{da}{dx}=2x\\exp{x^2},\n",
@@ -1589,9 +1379,7 @@
{
"cell_type": "markdown",
"id": "1ed5c66b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"as before."
]
@@ -1599,9 +1387,7 @@
{
"cell_type": "markdown",
"id": "6157ca77",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Forward and reverse modes\n",
"\n",
@@ -1611,9 +1397,7 @@
{
"cell_type": "markdown",
"id": "d47f8037",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{df}{dx}=\\frac{df}{db}\\frac{db}{da}\\frac{da}{dx},\n",
@@ -1623,9 +1407,7 @@
{
"cell_type": "markdown",
"id": "7a7bbc5d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which we can rewrite either as"
]
@@ -1633,9 +1415,7 @@
{
"cell_type": "markdown",
"id": "427cb024",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{df}{dx}=\\left[\\frac{df}{db}\\frac{db}{da}\\right]\\frac{da}{dx},\n",
@@ -1645,9 +1425,7 @@
{
"cell_type": "markdown",
"id": "57a1aaa6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"or"
]
@@ -1655,9 +1433,7 @@
{
"cell_type": "markdown",
"id": "a8faabb8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{df}{dx}=\\frac{df}{db}\\left[\\frac{db}{da}\\frac{da}{dx}\\right].\n",
@@ -1667,9 +1443,7 @@
{
"cell_type": "markdown",
"id": "bb053125",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The first expression is called reverse mode (or back propagation)\n",
"since we start by evaluating the derivatives at the end point and then\n",
@@ -1685,9 +1459,7 @@
{
"cell_type": "markdown",
"id": "6d58682a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More complicated function\n",
"\n",
@@ -1697,9 +1469,7 @@
{
"cell_type": "markdown",
"id": "e26f3e58",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x) =\\sqrt{x^2+exp{x^2}},\n",
@@ -1709,9 +1479,7 @@
{
"cell_type": "markdown",
"id": "bd6d2ab5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with derivative"
]
@@ -1719,9 +1487,7 @@
{
"cell_type": "markdown",
"id": "4df7d922",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f'(x) =\\frac{x(1+\\exp{x^2})}{\\sqrt{x^2+exp{x^2}}}.\n",
@@ -1731,9 +1497,7 @@
{
"cell_type": "markdown",
"id": "451135b1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The corresponding SymPy code reads"
]
@@ -1742,10 +1506,7 @@
"cell_type": "code",
"execution_count": 5,
"id": "a0c4f830",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"from __future__ import division\n",
@@ -1761,9 +1522,7 @@
{
"cell_type": "markdown",
"id": "dd8fb453",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Counting the number of floating point operations\n",
"\n",
@@ -1777,9 +1536,7 @@
{
"cell_type": "markdown",
"id": "4a4d0c85",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Defining intermediate operations\n",
"\n",
@@ -1790,9 +1547,7 @@
{
"cell_type": "markdown",
"id": "6216f527",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"a = x^2,\n",
@@ -1802,9 +1557,7 @@
{
"cell_type": "markdown",
"id": "36fa3364",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1812,9 +1565,7 @@
{
"cell_type": "markdown",
"id": "d7936338",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b = \\exp{x^2} = \\exp{a},\n",
@@ -1824,9 +1575,7 @@
{
"cell_type": "markdown",
"id": "b3c1cf2e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1834,9 +1583,7 @@
{
"cell_type": "markdown",
"id": "3045f2f3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"c= a+b,\n",
@@ -1846,9 +1593,7 @@
{
"cell_type": "markdown",
"id": "c62d9b4f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1856,9 +1601,7 @@
{
"cell_type": "markdown",
"id": "cc00424b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"d=f(x)=\\sqrt{c}.\n",
@@ -1868,9 +1611,7 @@
{
"cell_type": "markdown",
"id": "2880b26d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## New expression for the derivative\n",
"\n",
@@ -1880,9 +1621,7 @@
{
"cell_type": "markdown",
"id": "6995073b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial a}{\\partial x} = 2x,\n",
@@ -1892,9 +1631,7 @@
{
"cell_type": "markdown",
"id": "3025c95c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1902,9 +1639,7 @@
{
"cell_type": "markdown",
"id": "eadbab8c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial b}{\\partial a} = \\exp{a},\n",
@@ -1914,9 +1649,7 @@
{
"cell_type": "markdown",
"id": "3b92e291",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1924,9 +1657,7 @@
{
"cell_type": "markdown",
"id": "61abe762",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial c}{\\partial a} = 1,\n",
@@ -1936,9 +1667,7 @@
{
"cell_type": "markdown",
"id": "b9ed2e77",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1946,9 +1675,7 @@
{
"cell_type": "markdown",
"id": "3dc26776",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial c}{\\partial b} = 1,\n",
@@ -1958,9 +1685,7 @@
{
"cell_type": "markdown",
"id": "8092c5dc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -1968,9 +1693,7 @@
{
"cell_type": "markdown",
"id": "3408773c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial d}{\\partial c} = \\frac{1}{2\\sqrt{c}},\n",
@@ -1980,9 +1703,7 @@
{
"cell_type": "markdown",
"id": "547d309c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and finally"
]
@@ -1990,9 +1711,7 @@
{
"cell_type": "markdown",
"id": "3885960f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial d} = 1.\n",
@@ -2002,9 +1721,7 @@
{
"cell_type": "markdown",
"id": "094da560",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final derivatives\n",
"Our final derivatives are thus"
@@ -2013,9 +1730,7 @@
{
"cell_type": "markdown",
"id": "5e7784ef",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial c} = \\frac{\\partial f}{\\partial d} \\frac{\\partial d}{\\partial c} = \\frac{1}{2\\sqrt{c}},\n",
@@ -2025,9 +1740,7 @@
{
"cell_type": "markdown",
"id": "da55dc29",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial b} = \\frac{\\partial f}{\\partial c} \\frac{\\partial c}{\\partial b} = \\frac{1}{2\\sqrt{c}},\n",
@@ -2037,9 +1750,7 @@
{
"cell_type": "markdown",
"id": "1521da52",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial a} = \\frac{\\partial f}{\\partial c} \\frac{\\partial c}{\\partial a}+\n",
@@ -2050,9 +1761,7 @@
{
"cell_type": "markdown",
"id": "c585cfc1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and finally"
]
@@ -2060,9 +1769,7 @@
{
"cell_type": "markdown",
"id": "49baac68",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial x} = \\frac{\\partial f}{\\partial a} \\frac{\\partial a}{\\partial x} = \\frac{x(1+\\exp{a})}{\\sqrt{c}},\n",
@@ -2072,9 +1779,7 @@
{
"cell_type": "markdown",
"id": "90c1b377",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which is just"
]
@@ -2082,9 +1787,7 @@
{
"cell_type": "markdown",
"id": "dfd1e5f5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial x} = \\frac{x(1+b)}{d},\n",
@@ -2094,9 +1797,7 @@
{
"cell_type": "markdown",
"id": "01027588",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and requires only three operations if we can reuse all intermediate variables."
]
@@ -2104,9 +1805,7 @@
{
"cell_type": "markdown",
"id": "faf9a7f4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## In general not this simple\n",
"\n",
@@ -2120,9 +1819,7 @@
{
"cell_type": "markdown",
"id": "c8a60c3c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Automatic differentiation\n",
"\n",
@@ -2137,9 +1834,7 @@
{
"cell_type": "markdown",
"id": "bebf89c4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{bmatrix} x_1=x & x_2 = x^2=a & x_3 =\\exp{a}= b & x_4=c=a+b & x_5 = \\sqrt{c}=d \\end{bmatrix}.\n",
@@ -2149,9 +1844,7 @@
{
"cell_type": "markdown",
"id": "c8e55af0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Furthemore, for $i=l+1, \\dots, L$ (here $i=2,3,4,5$ and $f=x_L=d$), we\n",
"define the elementary functions $g_i(x_{Pa(x_i)})$ where $x_{Pa(x_i)}$ are the parent nodes of the variable $x_i$.\n",
@@ -2162,9 +1855,7 @@
{
"cell_type": "markdown",
"id": "4b69aacb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Chain rule\n",
"\n",
@@ -2175,9 +1866,7 @@
{
"cell_type": "markdown",
"id": "c0d54321",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial x_L} = 1,\n",
@@ -2187,9 +1876,7 @@
{
"cell_type": "markdown",
"id": "8486d91e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which allows us to find the derivatives of the various variables $x_i$ as"
]
@@ -2197,9 +1884,7 @@
{
"cell_type": "markdown",
"id": "de5e8a70",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f}{\\partial x_i} = \\sum_{x_j:x_i\\in Pa(x_j)}\\frac{\\partial f}{\\partial x_j} \\frac{\\partial x_j}{\\partial x_i}=\\sum_{x_j:x_i\\in Pa(x_j)}\\frac{\\partial f}{\\partial x_j} \\frac{\\partial g_j}{\\partial x_i}.\n",
@@ -2209,9 +1894,7 @@
{
"cell_type": "markdown",
"id": "42dc3f26",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Whenever we have a function which can be expressed as a computation\n",
"graph and the various functions can be expressed in terms of\n",
@@ -2224,9 +1907,7 @@
{
"cell_type": "markdown",
"id": "133e16c7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## First network example, simple percepetron with one input\n",
"\n",
@@ -2239,9 +1920,7 @@
{
"cell_type": "markdown",
"id": "1fd248a5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_1 = w_1x+b_1,\n",
@@ -2251,9 +1930,7 @@
{
"cell_type": "markdown",
"id": "321220a1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $w_1$ is the weight and $b_1$ is the bias. These are the\n",
"parameters we want to optimize. The output is $a_1=\\sigma(z_1)$ (see\n",
@@ -2265,9 +1942,7 @@
{
"cell_type": "markdown",
"id": "d4ef8cda",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C(x;w_1,b_1)=\\frac{1}{2}(a_1-y)^2.\n",
@@ -2277,9 +1952,7 @@
{
"cell_type": "markdown",
"id": "c83257a3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Layout of a simple neural network with no hidden layer\n",
"\n",
@@ -2293,9 +1966,7 @@
{
"cell_type": "markdown",
"id": "41976e3b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Optimizing the parameters\n",
"\n",
@@ -2309,9 +1980,7 @@
{
"cell_type": "markdown",
"id": "daeda85f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_1} \\hspace{0.1cm}\\mathrm{and}\\hspace{0.1cm}\\frac{\\partial C}{\\partial b_1}.\n",
@@ -2321,9 +1990,7 @@
{
"cell_type": "markdown",
"id": "cfd25fda",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Using the chain rule we find"
]
@@ -2331,9 +1998,7 @@
{
"cell_type": "markdown",
"id": "8d695dcf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_1-y)\\sigma_1'x,\n",
@@ -2343,9 +2008,7 @@
{
"cell_type": "markdown",
"id": "dcc2f332",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -2353,9 +2016,7 @@
{
"cell_type": "markdown",
"id": "cb53a1ce",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_1-y)\\sigma_1',\n",
@@ -2365,9 +2026,7 @@
{
"cell_type": "markdown",
"id": "b547af4b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which we later will just define as"
]
@@ -2375,9 +2034,7 @@
{
"cell_type": "markdown",
"id": "e05ad1a5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}=\\delta_1.\n",
@@ -2387,9 +2044,7 @@
{
"cell_type": "markdown",
"id": "e841d792",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Adding a hidden layer\n",
"\n",
@@ -2403,9 +2058,7 @@
{
"cell_type": "markdown",
"id": "7d0be7a7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_1 = w_1a_0+b_1 \\hspace{0.1cm} \\wedge a_1 = \\sigma_1(z_1),\n",
@@ -2415,9 +2068,7 @@
{
"cell_type": "markdown",
"id": "99d7e794",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_2 = w_2a_1+b_2 \\hspace{0.1cm} \\wedge a_2 = \\sigma_2(z_2),\n",
@@ -2427,9 +2078,7 @@
{
"cell_type": "markdown",
"id": "b5a9ccd1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and the cost function"
]
@@ -2437,9 +2086,7 @@
{
"cell_type": "markdown",
"id": "97e1f77a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C(x;\\boldsymbol{\\Theta})=\\frac{1}{2}(a_2-y)^2,\n",
@@ -2449,9 +2096,7 @@
{
"cell_type": "markdown",
"id": "129d6040",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $\\boldsymbol{\\Theta}=[w_1,w_2,b_1,b_2]$."
]
@@ -2459,9 +2104,7 @@
{
"cell_type": "markdown",
"id": "cd905c05",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Layout of a simple neural network with one hidden layer\n",
"\n",
@@ -2475,9 +2118,7 @@
{
"cell_type": "markdown",
"id": "dc3ee985",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The derivatives\n",
"\n",
@@ -2487,9 +2128,7 @@
{
"cell_type": "markdown",
"id": "6b21de52",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial w_2}=(a_2-y)\\sigma_2'a_1=\\delta_2a_1,\n",
@@ -2499,9 +2138,7 @@
{
"cell_type": "markdown",
"id": "04bef693",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial b_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial b_2}=(a_2-y)\\sigma_2'=\\delta_2,\n",
@@ -2511,9 +2148,7 @@
{
"cell_type": "markdown",
"id": "6e9a9575",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_2-y)\\sigma_2'a_1\\sigma_1'a_0,\n",
@@ -2523,9 +2158,7 @@
{
"cell_type": "markdown",
"id": "c65ff641",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_2-y)\\sigma_2'\\sigma_1'=\\delta_1.\n",
@@ -2535,9 +2168,7 @@
{
"cell_type": "markdown",
"id": "f67e1353",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Can you generalize this to more than one hidden layer?"
]
@@ -2545,9 +2176,7 @@
{
"cell_type": "markdown",
"id": "7476ad06",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Important observations\n",
"\n",
@@ -2561,9 +2190,7 @@
{
"cell_type": "markdown",
"id": "520d8a90",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The training\n",
"\n",
@@ -2573,9 +2200,7 @@
{
"cell_type": "markdown",
"id": "d57d0f1d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{i}\\leftarrow w_{i}- \\eta \\delta_i a_{i-1},\n",
@@ -2585,9 +2210,7 @@
{
"cell_type": "markdown",
"id": "97925a47",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -2595,9 +2218,7 @@
{
"cell_type": "markdown",
"id": "21bf2445",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b_i \\leftarrow b_i-\\eta \\delta_i,\n",
@@ -2607,9 +2228,7 @@
{
"cell_type": "markdown",
"id": "de7f0e7b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $\\eta$ is the learning rate.\n",
"\n",
@@ -2621,9 +2240,7 @@
{
"cell_type": "markdown",
"id": "4060cfa4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Code example\n",
"\n",
@@ -2639,10 +2256,7 @@
"cell_type": "code",
"execution_count": 6,
"id": "1248e89b",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
@@ -2712,9 +2326,7 @@
{
"cell_type": "markdown",
"id": "eabe6b42",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small."
]
@@ -2722,9 +2334,7 @@
{
"cell_type": "markdown",
"id": "35974e2d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Exercise 1: Including more data\n",
"\n",
@@ -2740,9 +2350,7 @@
{
"cell_type": "markdown",
"id": "eebf9d30",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Simple neural network and the back propagation equations\n",
"\n",
@@ -2757,9 +2365,7 @@
{
"cell_type": "markdown",
"id": "2f079f0b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"x_0 = a_0^{(0)} \\wedge x_1 = a_1^{(0)}.\n",
@@ -2769,9 +2375,7 @@
{
"cell_type": "markdown",
"id": "50b799fe",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The hidden layer (layer $(1)$) has nodes which yield the outputs $a_0^{(1)}$ and $a_1^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters"
]
@@ -2779,9 +2383,7 @@
{
"cell_type": "markdown",
"id": "9eeeddd7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{ij}^{(1)}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_0^{(1)},b_1^{(1)}\\right\\}.\n",
@@ -2791,9 +2393,7 @@
{
"cell_type": "markdown",
"id": "89d239c6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Layout of a simple neural network with two input nodes, one hidden layer and one output node\n",
"\n",
@@ -2807,9 +2407,7 @@
{
"cell_type": "markdown",
"id": "bdbf35b5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The ouput layer\n",
"\n",
@@ -2819,9 +2417,7 @@
{
"cell_type": "markdown",
"id": "4e4ff5e9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{i}^{(2)}=\\left\\{w_{0}^{(2)},w_{1}^{(2)}\\right\\} \\wedge b^{(2)}.\n",
@@ -2831,9 +2427,7 @@
{
"cell_type": "markdown",
"id": "0d262c6f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Our output is $\\tilde{y}=a^{(2)}$ and we define a generic cost function $C(a^{(2)},y;\\boldsymbol{\\Theta})$ where $y$ is the target value (a scalar here).\n",
"The parameters we need to optimize are given by"
@@ -2842,9 +2436,7 @@
{
"cell_type": "markdown",
"id": "bc03ac5a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Theta}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)},w_{0}^{(2)},w_{1}^{(2)},b_0^{(1)},b_1^{(1)},b^{(2)}\\right\\}.\n",
@@ -2854,9 +2446,7 @@
{
"cell_type": "markdown",
"id": "59f909e2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Compact expressions\n",
"\n",
@@ -2867,9 +2457,7 @@
{
"cell_type": "markdown",
"id": "7eb0a44d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{bmatrix}z_0^{(1)} \\\\ z_1^{(1)} \\end{bmatrix}=\\begin{bmatrix}w_{00}^{(1)} & w_{01}^{(1)}\\\\ w_{10}^{(1)} &w_{11}^{(1)} \\end{bmatrix}\\begin{bmatrix}a_0^{(0)} \\\\ a_1^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_0^{(1)} \\\\ b_1^{(1)} \\end{bmatrix},\n",
@@ -2879,9 +2467,7 @@
{
"cell_type": "markdown",
"id": "07d8d778",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with outputs"
]
@@ -2889,9 +2475,7 @@
{
"cell_type": "markdown",
"id": "60368bf9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{bmatrix}a_0^{(1)} \\\\ a_1^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_0^{(1)}) \\\\ \\sigma^{(1)}(z_1^{(1)}) \\end{bmatrix}.\n",
@@ -2901,9 +2485,7 @@
{
"cell_type": "markdown",
"id": "79853185",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Output layer\n",
"\n",
@@ -2913,9 +2495,7 @@
{
"cell_type": "markdown",
"id": "e84489fb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z^{(2)} = w_{0}^{(2)}a_0^{(1)} +w_{1}^{(2)}a_1^{(1)}+b^{(2)},\n",
@@ -2925,9 +2505,7 @@
{
"cell_type": "markdown",
"id": "10ac6ea6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"resulting in the output"
]
@@ -2935,9 +2513,7 @@
{
"cell_type": "markdown",
"id": "4bb215d6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"a^{(2)}=\\sigma^{(2)}(z^{(2)}).\n",
@@ -2947,9 +2523,7 @@
{
"cell_type": "markdown",
"id": "0389c02c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Explicit derivatives\n",
"\n",
@@ -2963,9 +2537,7 @@
{
"cell_type": "markdown",
"id": "bd9a8665",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_{i}^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial w_{i}^{(2)}}=\\delta^{(2)}a_i^{(1)},\n",
@@ -2975,9 +2547,7 @@
{
"cell_type": "markdown",
"id": "904eef6d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with"
]
@@ -2985,9 +2555,7 @@
{
"cell_type": "markdown",
"id": "f422cd5d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta^{(2)}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n",
@@ -2997,9 +2565,7 @@
{
"cell_type": "markdown",
"id": "0827f570",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and finally"
]
@@ -3007,9 +2573,7 @@
{
"cell_type": "markdown",
"id": "0caae8eb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial b^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial b^{(2)}}=\\delta^{(2)}.\n",
@@ -3019,9 +2583,7 @@
{
"cell_type": "markdown",
"id": "2f4011e7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Derivatives of the hidden layer\n",
"\n",
@@ -3031,9 +2593,7 @@
{
"cell_type": "markdown",
"id": "4614eb63",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n",
@@ -3044,9 +2604,7 @@
{
"cell_type": "markdown",
"id": "954403b8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which, noting that"
]
@@ -3054,9 +2612,7 @@
{
"cell_type": "markdown",
"id": "d0f5c05f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z^{(2)} =w_0^{(2)}a_0^{(1)}+w_1^{(2)}a_1^{(1)}+b^{(2)},\n",
@@ -3066,9 +2622,7 @@
{
"cell_type": "markdown",
"id": "ac8c2f66",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"allows us to rewrite"
]
@@ -3076,9 +2630,7 @@
{
"cell_type": "markdown",
"id": "9bcf7c0d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}a_0^{(1)}.\n",
@@ -3088,9 +2640,7 @@
{
"cell_type": "markdown",
"id": "4c2c5c00",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final expression\n",
"Defining"
@@ -3099,9 +2649,7 @@
{
"cell_type": "markdown",
"id": "eed00134",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_0^{(1)}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}\\delta^{(2)},\n",
@@ -3111,9 +2659,7 @@
{
"cell_type": "markdown",
"id": "21d53946",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"we have"
]
@@ -3121,9 +2667,7 @@
{
"cell_type": "markdown",
"id": "4fbe8642",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\delta_0^{(1)}a_0^{(1)}.\n",
@@ -3133,9 +2677,7 @@
{
"cell_type": "markdown",
"id": "2325e21d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Similarly, we obtain"
]
@@ -3143,9 +2685,7 @@
{
"cell_type": "markdown",
"id": "14edd073",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_{01}^{(1)}}=\\delta_0^{(1)}a_1^{(1)}.\n",
@@ -3155,9 +2695,7 @@
{
"cell_type": "markdown",
"id": "52bbf902",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Completing the list\n",
"\n",
@@ -3167,9 +2705,7 @@
{
"cell_type": "markdown",
"id": "98a6ed3a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_{10}^{(1)}}=\\delta_1^{(1)}a_0^{(1)},\n",
@@ -3179,9 +2715,7 @@
{
"cell_type": "markdown",
"id": "6ab03495",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -3189,9 +2723,7 @@
{
"cell_type": "markdown",
"id": "dcda9d7d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)},\n",
@@ -3201,9 +2733,7 @@
{
"cell_type": "markdown",
"id": "46ec3534",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where we have defined"
]
@@ -3211,9 +2741,7 @@
{
"cell_type": "markdown",
"id": "f10c20da",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)}.\n",
@@ -3223,9 +2751,7 @@
{
"cell_type": "markdown",
"id": "c3c477be",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final expressions for the biases of the hidden layer\n",
"\n",
@@ -3235,9 +2761,7 @@
{
"cell_type": "markdown",
"id": "0c915c97",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial b_{0}^{(1)}}=\\delta_0^{(1)},\n",
@@ -3247,9 +2771,7 @@
{
"cell_type": "markdown",
"id": "646d8351",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -3257,9 +2779,7 @@
{
"cell_type": "markdown",
"id": "4ce93a76",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)}.\n",
@@ -3269,9 +2789,7 @@
{
"cell_type": "markdown",
"id": "ee7a38ed",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"As we will see below, these expressions can be generalized in a more compact form."
]
@@ -3279,9 +2797,7 @@
{
"cell_type": "markdown",
"id": "abdb541d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Gradient expressions\n",
"\n",
@@ -3292,9 +2808,7 @@
{
"cell_type": "markdown",
"id": "410de8d4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{i}^{(2)}\\leftarrow w_{i}^{(2)}- \\eta \\delta^{(2)} a_{i}^{(1)},\n",
@@ -3304,9 +2818,7 @@
{
"cell_type": "markdown",
"id": "ca1d6736",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -3314,9 +2826,7 @@
{
"cell_type": "markdown",
"id": "d43c2266",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b^{(2)} \\leftarrow b^{(2)}-\\eta \\delta^{(2)},\n",
@@ -3326,9 +2836,7 @@
{
"cell_type": "markdown",
"id": "56457d52",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -3336,9 +2844,7 @@
{
"cell_type": "markdown",
"id": "1ddc19f1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{ij}^{(1)}\\leftarrow w_{ij}^{(1)}- \\eta \\delta_{i}^{(1)} a_{j}^{(0)},\n",
@@ -3348,9 +2854,7 @@
{
"cell_type": "markdown",
"id": "de8c80cd",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -3358,9 +2862,7 @@
{
"cell_type": "markdown",
"id": "db88453a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b_{i}^{(1)} \\leftarrow b_{i}^{(1)}-\\eta \\delta_{i}^{(1)},\n",
@@ -3370,9 +2872,7 @@
{
"cell_type": "markdown",
"id": "f61958e4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $\\eta$ is the learning rate."
]
@@ -3380,9 +2880,7 @@
{
"cell_type": "markdown",
"id": "7ef82615",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Exercise 2: Extended program\n",
"\n",
@@ -3392,9 +2890,7 @@
{
"cell_type": "markdown",
"id": "f65323bf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"y=f(x_0,x_1)=x_0^2+3x_0x_1+x_1^2+5.\n",
@@ -3404,9 +2900,7 @@
{
"cell_type": "markdown",
"id": "96734d00",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We feed our network with $n=100$ entries $x_0$ and $x_1$. We have thus two features represented by these variable and an input matrix/design matrix $\\boldsymbol{X}\\in \\mathbf{R}^{n\\times 2}$"
]
@@ -3414,9 +2908,7 @@
{
"cell_type": "markdown",
"id": "f776f53c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} x_{00} & x_{01} \\\\ x_{00} & x_{01} \\\\ x_{10} & x_{11} \\\\ x_{20} & x_{21} \\\\ \\dots & \\dots \\\\ \\dots & \\dots \\\\ x_{n-20} & x_{n-21} \\\\ x_{n-10} & x_{n-11} \\end{bmatrix}.\n",
@@ -3426,9 +2918,7 @@
{
"cell_type": "markdown",
"id": "bbf2e007",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Write a code, based on the previous code examples, which takes as input these data and fit the above function.\n",
"You can extend your code to include automatic differentiation.\n",
@@ -3439,9 +2929,7 @@
{
"cell_type": "markdown",
"id": "9d6d0a32",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Getting serious, the back propagation equations for a neural network\n",
"\n",
@@ -3453,9 +2941,7 @@
{
"cell_type": "markdown",
"id": "6720dd98",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}((\\boldsymbol{\\Theta}^L)}{\\partial w_{jk}^L} = \\left(a_j^L - y_j\\right)a_j^L(1-a_j^L)a_k^{L-1},\n",
@@ -3465,9 +2951,7 @@
{
"cell_type": "markdown",
"id": "759eb99f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Defining"
]
@@ -3475,9 +2959,7 @@
{
"cell_type": "markdown",
"id": "294cc167",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L = a_j^L(1-a_j^L)\\left(a_j^L - y_j\\right) = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n",
@@ -3487,9 +2969,7 @@
{
"cell_type": "markdown",
"id": "3311576a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and using the Hadamard product of two vectors we can write this as"
]
@@ -3497,9 +2977,7 @@
{
"cell_type": "markdown",
"id": "892ab641",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\delta}^L = \\sigma'(\\hat{z}^L)\\circ\\frac{\\partial {\\cal C}}{\\partial (\\boldsymbol{a}^L)}.\n",
@@ -3509,9 +2987,7 @@
{
"cell_type": "markdown",
"id": "e126a949",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Analyzing the last results\n",
"\n",
@@ -3527,9 +3003,7 @@
{
"cell_type": "markdown",
"id": "0640b440",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More considerations\n",
"\n",
@@ -3545,9 +3019,7 @@
{
"cell_type": "markdown",
"id": "780544e8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}\n",
@@ -3557,9 +3029,7 @@
{
"cell_type": "markdown",
"id": "5ed8c781",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"With the definition of $\\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely"
]
@@ -3567,9 +3037,7 @@
{
"cell_type": "markdown",
"id": "9a9568a1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial{\\cal C}}{\\partial w_{jk}^L} = \\delta_j^La_k^{L-1}.\n",
@@ -3579,9 +3047,7 @@
{
"cell_type": "markdown",
"id": "29ac7458",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Derivatives in terms of $z_j^L$\n",
"\n",
@@ -3591,9 +3057,7 @@
{
"cell_type": "markdown",
"id": "693c53ac",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L =\\frac{\\partial {\\cal C}}{\\partial z_j^L}= \\frac{\\partial {\\cal C}}{\\partial a_j^L}\\frac{\\partial a_j^L}{\\partial z_j^L},\n",
@@ -3603,9 +3067,7 @@
{
"cell_type": "markdown",
"id": "650e4dea",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely"
]
@@ -3613,9 +3075,7 @@
{
"cell_type": "markdown",
"id": "4fbf17cf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L}\\frac{\\partial b_j^L}{\\partial z_j^L}=\\frac{\\partial {\\cal C}}{\\partial b_j^L},\n",
@@ -3625,9 +3085,7 @@
{
"cell_type": "markdown",
"id": "a7869a22",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"That is, the error $\\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias."
]
@@ -3635,9 +3093,7 @@
{
"cell_type": "markdown",
"id": "cfd6c9d7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Bringing it together\n",
"\n",
@@ -3647,9 +3103,7 @@
{
"cell_type": "markdown",
"id": "73360f7e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3665,9 +3119,7 @@
{
"cell_type": "markdown",
"id": "2685bc86",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -3675,9 +3127,7 @@
{
"cell_type": "markdown",
"id": "ff174cf5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3693,9 +3143,7 @@
{
"cell_type": "markdown",
"id": "1c97902a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -3703,9 +3151,7 @@
{
"cell_type": "markdown",
"id": "d70e9292",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3721,9 +3167,7 @@
{
"cell_type": "markdown",
"id": "ffcecfbe",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final back propagating equation\n",
"\n",
@@ -3733,9 +3177,7 @@
{
"cell_type": "markdown",
"id": "e49c2a4e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l =\\frac{\\partial {\\cal C}}{\\partial z_j^l}.\n",
@@ -3745,9 +3187,7 @@
{
"cell_type": "markdown",
"id": "4c706310",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We want to express this in terms of the equations for layer $l+1$."
]
@@ -3755,9 +3195,7 @@
{
"cell_type": "markdown",
"id": "301cfb1f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Using the chain rule and summing over all $k$ entries\n",
"\n",
@@ -3767,9 +3205,7 @@
{
"cell_type": "markdown",
"id": "5dc4f3ba",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l =\\sum_k \\frac{\\partial {\\cal C}}{\\partial z_k^{l+1}}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}}=\\sum_k \\delta_k^{l+1}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}},\n",
@@ -3779,9 +3215,7 @@
{
"cell_type": "markdown",
"id": "3dfdac72",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and recalling that"
]
@@ -3789,9 +3223,7 @@
{
"cell_type": "markdown",
"id": "c04afa91",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_j^{l+1} = \\sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},\n",
@@ -3801,9 +3233,7 @@
{
"cell_type": "markdown",
"id": "b7a22173",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $M_l$ being the number of nodes in layer $l$, we obtain"
]
@@ -3811,9 +3241,7 @@
{
"cell_type": "markdown",
"id": "fa9ff041",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l =\\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n",
@@ -3823,9 +3251,7 @@
{
"cell_type": "markdown",
"id": "a1648ab9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This is our final equation.\n",
"\n",
@@ -3835,9 +3261,7 @@
{
"cell_type": "markdown",
"id": "24569ace",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the back propagation algorithm\n",
"\n",
@@ -3858,9 +3282,7 @@
{
"cell_type": "markdown",
"id": "478d8aec",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the back propagation algorithm, part 2\n",
"\n",
@@ -3870,9 +3292,7 @@
{
"cell_type": "markdown",
"id": "a17614e1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n",
@@ -3882,9 +3302,7 @@
{
"cell_type": "markdown",
"id": "2bc63b98",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Then we compute the back propagate error for each $l=L-1,L-2,\\dots,1$ as"
]
@@ -3892,9 +3310,7 @@
{
"cell_type": "markdown",
"id": "060a6a00",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n",
@@ -3904,9 +3320,7 @@
{
"cell_type": "markdown",
"id": "f73677e1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the Back propagation algorithm, part 3\n",
"\n",
@@ -3918,9 +3332,7 @@
{
"cell_type": "markdown",
"id": "b0faa8a5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n",
@@ -3930,9 +3342,7 @@
{
"cell_type": "markdown",
"id": "4f0820e6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
@@ -3942,9 +3352,7 @@
{
"cell_type": "markdown",
"id": "72ac45e2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $\\eta$ being the learning rate."
]
@@ -3952,9 +3360,7 @@
{
"cell_type": "markdown",
"id": "53cbf551",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Updating the gradients\n",
"\n",
@@ -3964,9 +3370,7 @@
{
"cell_type": "markdown",
"id": "8a9433ea",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}sigma'(z_j^l),\n",
@@ -3976,9 +3380,7 @@
{
"cell_type": "markdown",
"id": "07c92792",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,1$ and update the weights and biases according to the rules"
]
@@ -3986,9 +3388,7 @@
{
"cell_type": "markdown",
"id": "efa48fd7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n",
@@ -3998,9 +3398,7 @@
{
"cell_type": "markdown",
"id": "6c4db201",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
@@ -4010,9 +3408,7 @@
{
"cell_type": "markdown",
"id": "19ae4f4e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"### Activation functions\n",
"\n",
@@ -4033,9 +3429,7 @@
{
"cell_type": "markdown",
"id": "c478ce3c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"### Activation functions, Logistic and Hyperbolic ones\n",
"\n",
@@ -4052,9 +3446,7 @@
{
"cell_type": "markdown",
"id": "5142ca2b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x) = \\frac{1}{1 + e^{-x}},\n",
@@ -4064,9 +3456,7 @@
{
"cell_type": "markdown",
"id": "9c4a44fb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and the *hyperbolic tangent* function"
]
@@ -4074,9 +3464,7 @@
{
"cell_type": "markdown",
"id": "74fded03",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x) = \\tanh(x)\n",
@@ -4086,9 +3474,7 @@
{
"cell_type": "markdown",
"id": "fefab641",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"### Relevance\n",
"\n",
@@ -4103,10 +3489,7 @@
"cell_type": "code",
"execution_count": 7,
"id": "56c89538",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\"\"\"The sigmoid function (or the logistic curve) is a \n",
@@ -4185,9 +3568,7 @@
{
"cell_type": "markdown",
"id": "21bdb71c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Fine-tuning neural network hyperparameters\n",
"\n",
@@ -4214,9 +3595,7 @@
{
"cell_type": "markdown",
"id": "82072ebe",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Hidden layers\n",
"\n",
@@ -4240,9 +3619,7 @@
{
"cell_type": "markdown",
"id": "8bfa5b96",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Vanishing gradients\n",
"\n",
@@ -4262,9 +3639,7 @@
{
"cell_type": "markdown",
"id": "a52a67bd",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Exploding gradients\n",
"\n",
@@ -4280,9 +3655,7 @@
{
"cell_type": "markdown",
"id": "052308cd",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Is the Logistic activation function (Sigmoid) our choice?\n",
"\n",
@@ -4303,9 +3676,7 @@
{
"cell_type": "markdown",
"id": "2c943852",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Logistic function as the root of problems\n",
"\n",
@@ -4322,9 +3693,7 @@
{
"cell_type": "markdown",
"id": "09e98546",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The derivative of the Logistic funtion\n",
"\n",
@@ -4350,9 +3719,7 @@
{
"cell_type": "markdown",
"id": "89458b0d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Insights from the paper by Glorot and Bengio\n",
"\n",
@@ -4370,9 +3737,7 @@
{
"cell_type": "markdown",
"id": "bb787cf2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The RELU function family\n",
"\n",
@@ -4391,9 +3756,7 @@
{
"cell_type": "markdown",
"id": "8d6d8c18",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## ELU function\n",
"\n",
@@ -4405,9 +3768,7 @@
{
"cell_type": "markdown",
"id": "52c4e6b6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n",
@@ -4417,9 +3778,7 @@
{
"cell_type": "markdown",
"id": "d0dad2c7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Which activation function should we use?\n",
"\n",
@@ -4439,9 +3798,7 @@
{
"cell_type": "markdown",
"id": "31d1148a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More on activation functions, output layers\n",
"\n",
@@ -4461,9 +3818,7 @@
{
"cell_type": "markdown",
"id": "d6d621a7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Batch Normalization\n",
"\n",
@@ -4487,9 +3842,7 @@
{
"cell_type": "markdown",
"id": "379983d5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Dropout\n",
"\n",
@@ -4507,9 +3860,7 @@
{
"cell_type": "markdown",
"id": "b769aa51",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Gradient Clipping\n",
"\n",
@@ -4527,9 +3878,7 @@
{
"cell_type": "markdown",
"id": "abeef409",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## A top-down perspective on Neural networks\n",
"\n",
@@ -4553,9 +3902,7 @@
{
"cell_type": "markdown",
"id": "75165741",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More top-down perspectives\n",
"\n",
@@ -4582,9 +3929,7 @@
{
"cell_type": "markdown",
"id": "14f22927",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Limitations of supervised learning with deep networks\n",
"\n",
@@ -4600,9 +3945,7 @@
{
"cell_type": "markdown",
"id": "f5ad6d66",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Limitations of NNs\n",
"\n",
@@ -4616,9 +3959,7 @@
{
"cell_type": "markdown",
"id": "82f3c1c1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Homogeneous data\n",
"\n",
@@ -4628,9 +3969,7 @@
{
"cell_type": "markdown",
"id": "01fba9a0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More limitations\n",
"\n",
@@ -4640,7 +3979,25 @@
]
}
],
- "metadata": {},
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.18"
+ }
+ },
"nbformat": 4,
"nbformat_minor": 5
}