-
-
@@ -290,13 +344,13 @@ MathJax.Hub.Config({
-
-
Reading suggestions for both days: Aurelien Geron's chapter 10 and Hastie et al chapter 11. For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text. +For neural networks we recommend Goodfellow et al chapters 6 and 7 and Bishop 5.1-5.4
Overview Video, from Stochastic Gradient methods to Neural Networks
@@ -181,7 +182,7 @@ For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -201,7 +202,7 @@ $$
This in turn means that the gradient can be
@@ -223,7 +224,7 @@ minibatches. We denote these minibatches by \( B_k \) where
Thus a gradient descent step now looks like
@@ -270,12 +271,12 @@ the number of minibatches, as exemplified in the code below.
-
A natural question is when do we stop the search for a new minimum?
@@ -319,7 +320,7 @@ gave the lowest value.
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -341,7 +342,7 @@ function.
-
-
The stochastic gradient descent (SGD) is almost always used with a
@@ -487,7 +488,7 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
Let us try to get more intuition from these equations. It is helpful
@@ -523,7 +524,7 @@ $$
Notice that this equation is identical to previous one if we identify
@@ -583,7 +584,7 @@ One of the major advantages of NAG is that it allows for the use of a larger lea
In stochastic gradient descent, with and without momentum, we still
@@ -611,7 +612,7 @@ ADAM.
In RMS prop, in addition to keeping a running average of the first
@@ -644,7 +645,7 @@ learning rate for flat directions.
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
@@ -697,7 +698,7 @@ $$
Automatic differentiation (AD),
@@ -764,7 +765,7 @@ Using autograd we have
-
Here we
@@ -815,7 +816,7 @@ experiment with other, possibly more complicated, functions as well.
-
To differentiate with respect to two (or more) arguments of a Python
@@ -847,7 +848,7 @@ being differentiated with respect to.
-
-
-
-
-
-
-
@@ -1065,7 +1066,7 @@ Assigning a value to the variable being differentiated with respect to
-
-
-
-
Artificial neural networks are computational systems that can learn to
@@ -1154,7 +1155,7 @@ a weight variable.
The field of artificial neural networks has a long history of
@@ -1220,7 +1221,7 @@ humanities to life science and medicine.
An artificial neural network (ANN), is a computational model that
@@ -1247,7 +1248,7 @@ methods we discussed earlier.
The feed-forward neural network (FFNN) was the first and simplest type
@@ -1265,7 +1266,7 @@ to all nodes in the subsequent layer, making this a so-called
A different variant of FFNNs are convolutional neural networks
@@ -1291,7 +1292,7 @@ recognition.
So far we have only mentioned ANNs where information flows in one
@@ -1308,7 +1309,7 @@ especially well-suited for handwriting and speech recognition.
There are many other kinds of ANNs that have been developed. One type
@@ -1325,7 +1326,7 @@ type of NN due the unusual activation functions.
One uses often so-called fully-connected feed-forward neural networks
@@ -1339,7 +1340,7 @@ Such networks are often called multilayer perceptrons (MLPs).
According to the Universal approximation theorem, a feed-forward
@@ -1357,7 +1358,7 @@ as to not restrict the range of output values.
The output \( y \) is produced via the activation function \( f \)
@@ -1377,7 +1378,7 @@ of the outputs of all neurons in the previous layer.
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( z_i^1 \) of the input coordinates \( x_j \),
@@ -1430,7 +1431,7 @@ is obtained.
The output of neuron \( i \) in layer 2 is thus,
@@ -1463,7 +1464,7 @@ $$
We can generalize this expression to an MLP with \( l \) hidden
@@ -1485,7 +1486,7 @@ variables are the input values \( x_n \).
This confirms that an MLP, despite its quite convoluted mathematical
@@ -1517,7 +1518,7 @@ flexibility of a neural network.
We can introduce a more convenient notation for the activations in an A NN.
@@ -1559,7 +1560,7 @@ $$
The activation of node \( i \) in layer 2 is
@@ -1585,7 +1586,7 @@ used as input to the activation functions. For each operation
A property that characterizes a neural network, other than its
@@ -1608,7 +1609,7 @@ for a FFNN to fulfill the universal approximation theorem
The second requirement excludes all linear functions. Furthermore, in
@@ -1638,7 +1639,7 @@ $$
The sigmoid function are more biologically plausible because the
@@ -1650,7 +1651,7 @@ become the most popular for deep neural networks
-
The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of
@@ -1765,7 +1766,7 @@ like logistic regression or linear regression and their modifications on the oth
A neural network with only one layer, what we called the simple
@@ -1793,7 +1794,7 @@ the potential of being universal approximators.
As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
@@ -1826,7 +1827,7 @@ classes.
With our definition of the targets \( \hat{t} \), the outputs of the
@@ -1870,7 +1871,7 @@ $$
From the definition of the activation \( z_j^l \) we have
@@ -1898,7 +1899,7 @@ $$
With these definitions we can now compute the derivative of the cost function in terms of the weights.
@@ -1929,7 +1930,7 @@ $$
We have thus
@@ -1989,7 +1990,7 @@ $$
It is also easy to see that our previous equation can be written as
@@ -2012,14 +2013,14 @@ That is, the error \( \delta_j^L \) is exactly equal to the rate of change of th
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
We have that (replacing \( L \) with a general layer \( l \))
@@ -2116,7 +2117,7 @@ We are now ready to set up the algorithm for back propagation and learning the w
The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
diff --git a/doc/pub/week40/html/week40-solarized.html b/doc/pub/week40/html/week40-solarized.html
index 44c71f5ec..73fca6ac9 100644
--- a/doc/pub/week40/html/week40-solarized.html
+++ b/doc/pub/week40/html/week40-solarized.html
@@ -1,19 +1,19 @@
-
-
Overview Video, from Stochastic Gradient methods to Neural Networks
@@ -212,7 +267,7 @@ For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -230,7 +285,7 @@ $$
This in turn means that the gradient can be
@@ -250,7 +305,7 @@ minibatches. We denote these minibatches by \( B_k \) where
Thus a gradient descent step now looks like
@@ -293,12 +348,12 @@ the number of minibatches, as exemplified in the code below.
-
A natural question is when do we stop the search for a new minimum?
@@ -342,7 +397,7 @@ gave the lowest value.
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -362,7 +417,7 @@ function.
-
-
The stochastic gradient descent (SGD) is almost always used with a
@@ -503,7 +558,7 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
Let us try to get more intuition from these equations. It is helpful
@@ -533,7 +588,7 @@ $$
Notice that this equation is identical to previous one if we identify
@@ -589,7 +644,7 @@ One of the major advantages of NAG is that it allows for the use of a larger lea
In stochastic gradient descent, with and without momentum, we still
@@ -617,7 +672,7 @@ ADAM.
In RMS prop, in addition to keeping a running average of the first
@@ -648,7 +703,7 @@ learning rate for flat directions.
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
@@ -697,7 +752,7 @@ $$
Automatic differentiation (AD),
@@ -758,7 +813,7 @@ Using autograd we have
-
-
Here we
@@ -808,7 +863,7 @@ experiment with other, possibly more complicated, functions as well.
-
To differentiate with respect to two (or more) arguments of a Python
@@ -839,7 +894,7 @@ being differentiated with respect to.
-
-
-
-
-
-
-
-
@@ -1054,7 +1109,7 @@ Assigning a value to the variable being differentiated with respect to
-
-
-
-
Artificial neural networks are computational systems that can learn to
@@ -1141,7 +1196,7 @@ a weight variable.
The field of artificial neural networks has a long history of
@@ -1204,7 +1259,7 @@ humanities to life science and medicine.
An artificial neural network (ANN), is a computational model that
@@ -1231,7 +1286,7 @@ methods we discussed earlier.
The feed-forward neural network (FFNN) was the first and simplest type
@@ -1249,7 +1304,7 @@ to all nodes in the subsequent layer, making this a so-called
A different variant of FFNNs are convolutional neural networks
@@ -1275,7 +1330,7 @@ recognition.
So far we have only mentioned ANNs where information flows in one
@@ -1292,7 +1347,7 @@ especially well-suited for handwriting and speech recognition.
There are many other kinds of ANNs that have been developed. One type
@@ -1309,7 +1364,7 @@ type of NN due the unusual activation functions.
One uses often so-called fully-connected feed-forward neural networks
@@ -1323,7 +1378,7 @@ Such networks are often called multilayer perceptrons (MLPs).
According to the Universal approximation theorem, a feed-forward
@@ -1341,7 +1396,7 @@ as to not restrict the range of output values.
The output \( y \) is produced via the activation function \( f \)
@@ -1359,7 +1414,7 @@ of the outputs of all neurons in the previous layer.
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( z_i^1 \) of the input coordinates \( x_j \),
@@ -1406,7 +1461,7 @@ is obtained.
The output of neuron \( i \) in layer 2 is thus,
@@ -1435,7 +1490,7 @@ $$
We can generalize this expression to an MLP with \( l \) hidden
@@ -1455,7 +1510,7 @@ variables are the input values \( x_n \).
This confirms that an MLP, despite its quite convoluted mathematical
@@ -1484,7 +1539,7 @@ flexibility of a neural network.
We can introduce a more convenient notation for the activations in an A NN.
@@ -1523,7 +1578,7 @@ $$
The activation of node \( i \) in layer 2 is
@@ -1546,7 +1601,7 @@ used as input to the activation functions. For each operation
A property that characterizes a neural network, other than its
@@ -1563,7 +1618,7 @@ for a FFNN to fulfill the universal approximation theorem
The second requirement excludes all linear functions. Furthermore, in
@@ -1588,7 +1643,7 @@ $$
The sigmoid function are more biologically plausible because the
@@ -1600,7 +1655,7 @@ become the most popular for deep neural networks
-
The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of
@@ -1713,7 +1768,7 @@ like logistic regression or linear regression and their modifications on the oth
A neural network with only one layer, what we called the simple
@@ -1741,7 +1796,7 @@ the potential of being universal approximators.
As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
@@ -1772,7 +1827,7 @@ classes.
With our definition of the targets \( \hat{t} \), the outputs of the
@@ -1810,7 +1865,7 @@ $$
From the definition of the activation \( z_j^l \) we have
@@ -1832,7 +1887,7 @@ $$
With these definitions we can now compute the derivative of the cost function in terms of the weights.
@@ -1857,7 +1912,7 @@ $$
We have thus
@@ -1907,7 +1962,7 @@ $$
It is also easy to see that our previous equation can be written as
@@ -1924,14 +1979,14 @@ $$
That is, the error \( \delta_j^L \) is exactly equal to the rate of change of the cost function as a function of the bias.
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
$$
@@ -1985,7 +2040,7 @@ one \( L-1 \) in terms of the errors in the final output layer.
We have that (replacing \( L \) with a general layer \( l \))
@@ -2017,7 +2072,7 @@ We are now ready to set up the algorithm for back propagation and learning the w
The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
@@ -2091,7 +2146,7 @@ Here it is convenient to use stochastic gradient descent (see the examples below
-
-
Overview Video, from Stochastic Gradient methods to Neural Networks
@@ -217,7 +272,7 @@ For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -235,7 +290,7 @@ $$
This in turn means that the gradient can be
@@ -255,7 +310,7 @@ minibatches. We denote these minibatches by \( B_k \) where
Thus a gradient descent step now looks like
@@ -298,12 +353,12 @@ the number of minibatches, as exemplified in the code below.
-
A natural question is when do we stop the search for a new minimum?
@@ -347,7 +402,7 @@ gave the lowest value.
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -367,7 +422,7 @@ function.
-
-
The stochastic gradient descent (SGD) is almost always used with a
@@ -508,7 +563,7 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
Let us try to get more intuition from these equations. It is helpful
@@ -538,7 +593,7 @@ $$
Notice that this equation is identical to previous one if we identify
@@ -594,7 +649,7 @@ One of the major advantages of NAG is that it allows for the use of a larger lea
In stochastic gradient descent, with and without momentum, we still
@@ -622,7 +677,7 @@ ADAM.
In RMS prop, in addition to keeping a running average of the first
@@ -653,7 +708,7 @@ learning rate for flat directions.
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
@@ -702,7 +757,7 @@ $$
Automatic differentiation (AD),
@@ -763,7 +818,7 @@ Using autograd we have
-Stochastic Gradient Descent
+Stochastic Gradient Descent
Computation of gradients
+Computation of gradients
SGD example
+SGD example
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -249,7 +250,7 @@ $$
The gradient step
+The gradient step
Simple example code
+Simple example code
import numpy as np
+
import numpy as np
n = 100 #100 datapoints
M = 5 #size of each minibatch
@@ -302,7 +303,7 @@ all \( n \) datapoints.
When do we stop?
+When do we stop?
Slightly different approach
+Slightly different approach
import numpy as np
+
import numpy as np
def step_length(t,t0,t1):
return t0/(t+t1)
@@ -370,12 +371,12 @@ j = 0
Program for stochastic gradient
+Program for stochastic gradient
# Importing various packages
+
# Importing various packages
from math import exp, sqrt
from random import random, seed
import numpy as np
@@ -446,7 +447,7 @@ plt.show()
Momentum based GD
+Momentum based GD
More on momentum based approaches
+More on momentum based approaches
Momentum parameter
+Momentum parameter
Second moment of the gradient
+Second moment of the gradient
RMS prop
+RMS prop
ADAM optimizer
+ADAM optimizer
Practical tips
+Practical tips
Automatic differentiation
+Automatic differentiation
import autograd.numpy as np
+
import autograd.numpy as np
# To do elementwise differentiation:
from autograd import elementwise_grad as egrad
@@ -803,7 +804,7 @@ plt.show()
Using autograd
+Using autograd
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f1(x):
@@ -837,7 +838,7 @@ grad_analytical = 3*a**Autograd with more complicated functions
+
Autograd with more complicated functions
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f2(x1,x2):
return 3*x1**3 + x2*(x1 - 5) + 1
@@ -887,12 +888,12 @@ Note that the grad function will not produce the true gradient of the function.
More complicated functions using the elements of their arguments directly
+More complicated functions using the elements of their arguments directly
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f3(x): # Assumes x is an array of length 5 or higher
return 2*x[0] + 3*x[1] + 5*x[2] + 7*x[3] + 11*x[4]**2
@@ -921,12 +922,12 @@ could expect form a gradient-evaluting function.
Functions using mathematical functions from Numpy
+Functions using mathematical functions from Numpy
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f4(x):
return np.sqrt(1+x**2) + np.exp(x) + np.sin(2*np.pi*x)
@@ -948,12 +949,12 @@ f4_grad_analytical = x/np.sqrt(1 + x**
-
More autograd
+More autograd
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f5(x):
if x >= 0:
@@ -972,12 +973,12 @@ x = 2.7
And with loops
+And with loops
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f6_for(x):
val = 0
@@ -1005,7 +1006,7 @@ x = 0.5
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
# Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9
# The analytical derivative is: sum(i*x**(i-1))
@@ -1019,11 +1020,11 @@ f6_grad_analytical = 0
Using recursion
+Using recursion
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f7(n): # Assume that n is an integer
@@ -1057,7 +1058,7 @@ Note that if n is equal to zero or one, Autograd will give an error message. Thi
Unsupported functions
+Unsupported functions
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f8(x): # Assume x is an array
x[2] = 3
@@ -1083,11 +1084,11 @@ Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The
The syntax a.dot(b) when finding the dot product
+The syntax a.dot(b) when finding the dot product
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f9(a): # Assume a is an array with 2 elements
b = np.array([1.0,2.0])
@@ -1107,7 +1108,7 @@ which also computed the dot product can be used:
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f9_alternative(x): # Assume a is an array with 2 elements
b = np.array([1.0,2.0])
@@ -1126,12 +1127,12 @@ x = np.array([3.0,Recommended to avoid
+
Recommended to avoid
The documentation recommends to avoid inplace operations such as
a += b
+
a += b
a -= b
a*= b
a /=b
@@ -1140,7 +1141,7 @@ a /=b
Neural networks
+Neural networks
Artificial neurons
+Artificial neurons
Neural network types
+Neural network types
Feed-forward neural networks
+Feed-forward neural networks
Convolutional Neural Network
+Convolutional Neural Network
Recurrent neural networks
+Recurrent neural networks
Other types of networks
+Other types of networks
Multilayer perceptrons
+Multilayer perceptrons
Why multilayer perceptrons?
+Why multilayer perceptrons?
Mathematical model
+Mathematical model
Mathematical model
+Mathematical model
Mathematical model
+Mathematical model
Mathematical model
+Mathematical model
Mathematical model
+Mathematical model
Matrix-vector notation
+Matrix-vector notation
Matrix-vector notation and activation
+Matrix-vector notation and activation
Activation functions
+Activation functions
Activation functions, Logistic and Hyperbolic ones
+Activation functions, Logistic and Hyperbolic ones
Relevance
+Relevance
"""The sigmoid function (or the logistic curve) is a
+
"""The sigmoid function (or the logistic curve) is a
function that takes any real number, z, and outputs a number (0,1).
It is useful in neural networks for assigning weights on a relative scale.
The value z is the weighted sum of parameters involved in the learning algorithm."""
@@ -1726,7 +1727,7 @@ plt.show()
The multilayer perceptron (MLP)
+The multilayer perceptron (MLP)
From one to many layers, the universal approximation theorem
+From one to many layers, the universal approximation theorem
Deriving the back propagation code for a multilayer perceptron model
+Deriving the back propagation code for a multilayer perceptron model
Definitions
+Definitions
Derivatives and the chain rule
+Derivatives and the chain rule
Derivative of the cost function
+Derivative of the cost function
Bringing it together, first back propagation equation
+Bringing it together, first back propagation equation
Derivatives in terms of \( z_j^L \)
+Derivatives in terms of \( z_j^L \)
Bringing it together
+Bringing it together
$$
\begin{equation}
@@ -2076,7 +2077,7 @@ one \( L-1 \) in terms of the errors in the final output layer.
Final back propagating equation
+Final back propagating equation
Setting up the Back propagation algorithm
+Setting up the Back propagation algorithm
Oct 6, 2020
Oct 4, 2021
-Plan for week 40
+Plan for week 40
-
Reading suggestions for both days: Aurelien Geron's chapter 10 and Hastie et al chapter 11.
For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
+For neural networks we recommend Goodfellow et al chapters 6 and 7 and Bishop 5.1-5.4
-Overview video for week 40
+Overview video for week 40
-Stochastic Gradient Descent
+Stochastic Gradient Descent
-Computation of gradients
+Computation of gradients
-SGD example
+SGD example
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -274,7 +329,7 @@ $$
-The gradient step
+The gradient step
-Simple example code
+Simple example code
import numpy as np
+
import numpy as np
n = 100 #100 datapoints
M = 5 #size of each minibatch
@@ -325,7 +380,7 @@ all \( n \) datapoints.
-When do we stop?
+When do we stop?
-Slightly different approach
+Slightly different approach
import numpy as np
+
import numpy as np
def step_length(t,t0,t1):
return t0/(t+t1)
@@ -390,12 +445,12 @@ j = 0
-Program for stochastic gradient
+Program for stochastic gradient
# Importing various packages
+
# Importing various packages
from math import exp, sqrt
from random import random, seed
import numpy as np
@@ -466,7 +521,7 @@ plt.show()
-Momentum based GD
+Momentum based GD
-More on momentum based approaches
+More on momentum based approaches
-Momentum parameter
+Momentum parameter
-Second moment of the gradient
+Second moment of the gradient
-RMS prop
+RMS prop
-ADAM optimizer
+ADAM optimizer
-Practical tips
+Practical tips
-Automatic differentiation
+Automatic differentiation
import autograd.numpy as np
+
import autograd.numpy as np
# To do elementwise differentiation:
from autograd import elementwise_grad as egrad
@@ -796,7 +851,7 @@ plt.show()
Using autograd
+Using autograd
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f1(x):
@@ -829,7 +884,7 @@ grad_analytical = 3*a**Autograd with more complicated functions
+
Autograd with more complicated functions
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f2(x1,x2):
return 3*x1**3 + x2*(x1 - 5) + 1
@@ -879,12 +934,12 @@ Note that the grad function will not produce the true gradient of the function.
-More complicated functions using the elements of their arguments directly
+More complicated functions using the elements of their arguments directly
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f3(x): # Assumes x is an array of length 5 or higher
return 2*x[0] + 3*x[1] + 5*x[2] + 7*x[3] + 11*x[4]**2
@@ -913,12 +968,12 @@ could expect form a gradient-evaluting function.
Functions using mathematical functions from Numpy
+Functions using mathematical functions from Numpy
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f4(x):
return np.sqrt(1+x**2) + np.exp(x) + np.sin(2*np.pi*x)
@@ -939,12 +994,12 @@ f4_grad_analytical = x/np.sqrt(1 + x**
-More autograd
+More autograd
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f5(x):
if x >= 0:
@@ -962,12 +1017,12 @@ x = 2.7
-And with loops
+And with loops
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f6_for(x):
val = 0
@@ -995,7 +1050,7 @@ x = 0.5
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
# Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9
# The analytical derivative is: sum(i*x**(i-1))
@@ -1008,11 +1063,11 @@ f6_grad_analytical = 0
-Using recursion
+Using recursion
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f7(n): # Assume that n is an integer
@@ -1046,7 +1101,7 @@ Note that if n is equal to zero or one, Autograd will give an error message. Thi
-Unsupported functions
+Unsupported functions
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f8(x): # Assume x is an array
x[2] = 3
@@ -1072,11 +1127,11 @@ Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The
-The syntax a.dot(b) when finding the dot product
+The syntax a.dot(b) when finding the dot product
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f9(a): # Assume a is an array with 2 elements
b = np.array([1.0,2.0])
@@ -1096,7 +1151,7 @@ which also computed the dot product can be used:
import autograd.numpy as np
+
import autograd.numpy as np
from autograd import grad
def f9_alternative(x): # Assume a is an array with 2 elements
b = np.array([1.0,2.0])
@@ -1114,12 +1169,12 @@ x = np.array([3.0,Recommended to avoid
+
Recommended to avoid
The documentation recommends to avoid inplace operations such as
a += b
+
a += b
a -= b
a*= b
a /=b
@@ -1127,7 +1182,7 @@ a /=b
-Neural networks
+Neural networks
-Artificial neurons
+Artificial neurons
-Neural network types
+Neural network types
-Feed-forward neural networks
+Feed-forward neural networks
-Convolutional Neural Network
+Convolutional Neural Network
-Recurrent neural networks
+Recurrent neural networks
-Other types of networks
+Other types of networks
-Multilayer perceptrons
+Multilayer perceptrons
-Why multilayer perceptrons?
+Why multilayer perceptrons?
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Mathematical model
+Mathematical model
-Matrix-vector notation
+Matrix-vector notation
-Matrix-vector notation and activation
+Matrix-vector notation and activation
-Activation functions
+Activation functions
-Activation functions, Logistic and Hyperbolic ones
+Activation functions, Logistic and Hyperbolic ones
-Relevance
+Relevance
"""The sigmoid function (or the logistic curve) is a
+
"""The sigmoid function (or the logistic curve) is a
function that takes any real number, z, and outputs a number (0,1).
It is useful in neural networks for assigning weights on a relative scale.
The value z is the weighted sum of parameters involved in the learning algorithm."""
@@ -1675,7 +1730,7 @@ plt.show()
-The multilayer perceptron (MLP)
+The multilayer perceptron (MLP)
-From one to many layers, the universal approximation theorem
+From one to many layers, the universal approximation theorem
-Deriving the back propagation code for a multilayer perceptron model
+Deriving the back propagation code for a multilayer perceptron model
-Definitions
+Definitions
-Derivatives and the chain rule
+Derivatives and the chain rule
-Derivative of the cost function
+Derivative of the cost function
-Bringing it together, first back propagation equation
+Bringing it together, first back propagation equation
-Derivatives in terms of \( z_j^L \)
+Derivatives in terms of \( z_j^L \)
-Bringing it together
+Bringing it together
-Final back propagating equation
+Final back propagating equation
-Setting up the Back propagation algorithm
+Setting up the Back propagation algorithm
Oct 6, 2020
Oct 4, 2021
-Plan for week 40
+Plan for week 40
-
Reading suggestions for both days: Aurelien Geron's chapter 10 and Hastie et al chapter 11.
For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
+For neural networks we recommend Goodfellow et al chapters 6 and 7 and Bishop 5.1-5.4
-Overview video for week 40
+Overview video for week 40
-Stochastic Gradient Descent
+Stochastic Gradient Descent
-Computation of gradients
+Computation of gradients
-SGD example
+SGD example
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -279,7 +334,7 @@ $$
-The gradient step
+The gradient step
-Simple example code
+Simple example code
import numpy as np
+
import numpy as np
n = 100 #100 datapoints
M = 5 #size of each minibatch
@@ -330,7 +385,7 @@ all \( n \) datapoints.
-When do we stop?
+When do we stop?
-Slightly different approach
+Slightly different approach
import numpy as np
+
import numpy as np
def step_length(t,t0,t1):
return t0/(t+t1)
@@ -395,12 +450,12 @@ j = 0
-Program for stochastic gradient
+Program for stochastic gradient
# Importing various packages
+
# Importing various packages
from math import exp, sqrt
from random import random, seed
import numpy as np
@@ -471,7 +526,7 @@ plt.show()
-Momentum based GD
+Momentum based GD
-More on momentum based approaches
+More on momentum based approaches
-Momentum parameter
+Momentum parameter
-Second moment of the gradient
+Second moment of the gradient
-RMS prop
+RMS prop
-ADAM optimizer
+ADAM optimizer
-Practical tips
+Practical tips
-Automatic differentiation
+Automatic differentiation
import autograd.numpy as np
+