-
@@ -274,7 +275,7 @@ MathJax.Hub.Config({
We have that \( \mathrm{W}_l \) is an \( N_{l-1} \times N_l \) matrix, while \( \hat{b}_l \) and \( \hat{y}_l \) are \( N_l \times 1 \) column vectors. With this notation, the sum becomes a matrix-vector multiplication, and we can write -the equation for the activations of hidden layer 2 as +the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as $$ \begin{equation} \hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) = @@ -293,7 +294,7 @@ $$
-For an MLP there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units. +For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units. Hereafter we will call the various entities of a layer for nodes. There are also no connections within a single layer. @@ -292,7 +293,7 @@ like logistic regression or linear regression and their modifications on the oth
-With the activation function \( \hat{z}^l \) we can in turn define the +With the activation values \( \hat{z}^l \) we can in turn define the output of layer \( l \) as \( \hat{a}^l = f(\hat{z}^l) \) where \( f \) is our activation function. In the examples here we will use the sigmoid -function discussed in our logistic regression lectures and here as -well. We will also use the same activation function \( f \) for all layers +function discussed in our logistic regression lectures. We will also use the same activation function \( f \) for all layers and their nodes. It means we have $$ @@ -293,7 +293,7 @@ $$
This is an important expression. The second term on the right handside -measures how fast the cost is changing as a function of the $j$th +measures how fast the cost function is changing as a function of the $j$th output activation. If, for example, the cost function doesn't depend much on a particular output node \( j \), then \( \delta_j^L \) will be small, which is what we would expect. The first term on the right, measures @@ -264,9 +265,9 @@ Notice that everything in the above equations is easily computed. In particular, we compute \( z_j^L \) while computing the behaviour of the network, and it is only a small additional overhead to compute \( f'(z^L_j) \). The exact form of the derivative with respect to the -outpuwill, of course, depend on the form of the cost function. +output depends on the form of the cost function. However, provided the cost function is known there should be little -trouble computing +trouble in calculating $$ \frac{\partial {\cal C}}{\partial (a_j^L)} @@ -304,7 +305,7 @@ $$
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are + +
@@ -276,7 +279,7 @@ descent. In this case we say the system learns slowly.
Another interesting feature is that is when the activation function, -represented by the sigmoid function here, is rather flat when towards +represented by the sigmoid function here, is rather flat when we move towards its end values \( 0 \) and \( 1 \) (see the above Python codes). In these cases, the derivatives of the activation function will also be close to zero, meaning again that the gradients will be small and the @@ -314,7 +317,7 @@ one \( L-1 \) in terms of the errors in the final output layer.
The parameter \( \eta \) is the learning parameter discussed in connection with the gradient descent methods. -Here it is convenient to use stochastic radient descent with mini-batches with an outer loop that steps through multiple epochs of training. +Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.
@@ -330,7 +331,7 @@ Here it is convenient to use stochastic radient descent with mini-batches with a
For an input \( \boldsymbol{a} \) from the hidden layer, the probability that the input \( \boldsymbol{x} \) -is in class 0 or 1 is just: - +is in class 0 or 1 is just. We let \( \theta \) represent the unknown weights and biases to be adjusted by our equations). The variable \( x \) +represents our activation values \( z \). We have $$ -P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{a}^T \boldsymbol{w}_{out})} , +P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{x}} , $$ and @@ -292,7 +293,7 @@ of our network.
Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500 @@ -299,7 +300,7 @@ weights to the output layer.
The bias weights \( \boldsymbol{b} \) are often initialized to zero, but a small value like \( 0.01 \) ensures all neurons have some output which can be backpropagated in the first training cycle.
@@ -290,7 +291,7 @@ output_bias = np47
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
-For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer:
+For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer \( l \):
-$$ z_{j}^{h} = \sum_{i=1}^{F} w_{ij}^{h} x_i + b_{j}^{h} = \boldsymbol{x}^T \boldsymbol{w}_{j}^{h} + b_{j}^{h} ,$$
+$$ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$
this is then passed through our activation function
-$$ a_{j}^{h} = f(z_{j}^{h}) .$$
+$$ a_{j}^{l} = f(z_{j}^{l}) .$$
We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron \( j \) in the output layer:
-$$ z_{j}^{o} = \sum_{i=1}^{H} w_{ij}^{o} a_{i}^{h} + b_{j}^{o} = (\boldsymbol{a}^{h})^T \boldsymbol{w}_{j}^{o} + b_{j}^{o} .$$
+$$ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$
Finally we calculate the output of neuron \( j \) in the output layer using the softmax function:
-$$ a_{j}^{o} = \frac{\exp{(z_j^{o})}}
-{\sum_{c=0}^{C-1} \exp{(z_c^{o})}} .$$
+$$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
+{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .$$
@@ -281,7 +282,7 @@ $$ a_{j}^{o} = \frac{\exp{(z_j^{o})}}
meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image.
This is then passed through the activation:
-$$ A^{h} = f(Z^h) .$$
+$$ \hat{a}^{l} = f(\hat{z}^l) .$$
This is fed to the output layer:
-$$ Z^{o} = A^{h} W^{o} + B^{o} .$$
+$$ \hat{z}^{L} = \hat{a}^{L} \hat{W}^{L} + \hat{b}^{L} .$$
Finally we receive our output values for each image and each category by passing it through the softmax function:
-$$ output = softmax (Z^{o}) = (n_{inputs}, n_{categories}) .$$
+$$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
-
The gradient for the output weights is calculated as
-$$ \nabla W_{o} = A^T \Delta_o = (n_{hidden}, n_{categories}) ,$$
+$$ \nabla W_{L} = \hat{a}^T \delta_L = (n_{hidden}, n_{categories}) ,$$
-where \( A = (n_{inputs}, n_{hidden}) \). This simply means that we are summing up the gradients for each input.
+where \( \hat{a} = (n_{inputs}, n_{hidden}) \). This simply means that we are summing up the gradients for each input.
Since we are going backwards we have to transpose the activation matrix.
The gradient with respect to the output bias is then
-$$ \nabla B_{o} = \sum_{i=1}^{n_{inputs}} \Delta_o = (n_{categories}) .$$
+$$ \nabla \hat{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .$$
The error in the hidden layer is
-$$ \Delta_h = \Delta_o W_{o}^T \circ f'(Z_{h}) = \Delta_o W_{o}^T \circ A_{h} \circ (1 - A_{h}) = (n_{inputs}, n_{hidden}) ,$$
+$$ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$
-where \( f'(A_{h}) \) is the derivative of the activation in the hidden layer. The matrix products mean
+where \( f'(a_{h}) \) is the derivative of the activation in the hidden layer. The matrix products mean
that we are summing up the products for each neuron in the output layer. The symbol \( \circ \) denotes
the Hadamard product, meaning element-wise multiplication.
This again gives us the gradients in the hidden layer:
-$$ \nabla W_{h} = X^T \Delta_h = (n_{features}, n_{hidden}) ,$$
+$$ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,$$
-$$ \nabla B_{h} = \sum_{i=1}^{n_{inputs}} \Delta_h = (n_{hidden}) .$$
+$$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
@@ -371,7 +372,7 @@ lmbd = 0.0153
We now perform a grid search to find the optimal hyperparameters for the network.
-Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \( 98 \% \) (\( 2 \% \) error rate).
+Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \( 98\% \) (\( 2\% \) error rate).
-
-
-
-
-
-
-scikit-learn is a machine learning library for Python. It focuses more on traditional machine learning methods, such as regression, clustering, decision trees, etc. As such, it has only two types of neural networks: Multi Layer Perceptron outputting continuous values, MPLRegressor, and Multi Layer Perceptron outputting labels, MLPClassifier. We will see how simple it is to use these classes.
-
-
-scikit-learn implements a few improvements from our neural network, such as early stopping, a varying learning rate, different optimization methods, etc. We would therefore expect a better performance overall.
+
-
-
-
-
-Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
-and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy
-and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer.
+scikit-learn focuses more
+on traditional machine learning methods, such as regression,
+clustering, decision trees, etc. As such, it has only two types of
+neural networks: Multi Layer Perceptron outputting continuous values,
+MPLRegressor, and Multi Layer Perceptron outputting labels,
+MLPClassifier. We will see how simple it is to use these classes.
-In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite
-clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or
-NumPy arrays.
+scikit-learn implements a few improvements from our neural network,
+such as early stopping, a varying learning rate, different
+optimization methods, etc. We would therefore expect a better
+performance overall.
+
+
+
+
@@ -264,6 +289,8 @@ NumPy arrays.
-Tensorflow is an open source library machine learning library
-developed by the Google Brain team for internal use. It was released
-under the Apache 2.0 open source license in November 9, 2015.
-
-
-Tensorflow is a computational framework that allows you to construct
-machine learning models at different levels of abstraction, from
-high-level, object-oriented APIs like Keras, down to the C++ kernels
-that Tensorflow is built upon. The higher levels of abstraction are
-simpler to use, but less flexible, and our choice of implementation
-should reflect the problems we are trying to solve.
-
-
-Tensorflow uses so-called graphs to represent your computation
-in terms of the dependencies between individual operations, such that you first build a Tensorflow graph
-to represent your model, and then create a Tensorflow session to run the graph.
-
-
-In this guide we will analyze the same data as we did in our NumPy and
-scikit-learn tutorial, gathered from the MNIST database of images. We
-will give an introduction to the lower level Python Application
-Program Interfaces (APIs), and see how we use them to build our graph.
-Then we will build (effectively) the same graph in Keras, to see just
-how simple solving a machine learning problem can be.
-
-
-To install tensorflow on Unix/Linux systems, use pip as
+
-
-and/or if you use anaconda, just write (or install from the graphical user interface)
-
+
@@ -293,6 +291,8 @@ and/or if you use anaconda, just write (or install from the graphical use
+Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
+and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy
+and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer.
-
-
+In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite
+clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or
+NumPy arrays.
-
-
@@ -313,6 +263,8 @@ X_train, X_test, Y_train, Y_test = train_tes
+Tensorflow is an open source library machine learning library
+developed by the Google Brain team for internal use. It was released
+under the Apache 2.0 open source license in November 9, 2015.
+
+Tensorflow is a computational framework that allows you to construct
+machine learning models at different levels of abstraction, from
+high-level, object-oriented APIs like Keras, down to the C++ kernels
+that Tensorflow is built upon. The higher levels of abstraction are
+simpler to use, but less flexible, and our choice of implementation
+should reflect the problems we are trying to solve.
+
+
+Tensorflow uses so-called graphs to represent your computation
+in terms of the dependencies between individual operations, such that you first build a Tensorflow graph
+to represent your model, and then create a Tensorflow session to run the graph.
+
+
+In this guide we will analyze the same data as we did in our NumPy and
+scikit-learn tutorial, gathered from the MNIST database of images. We
+will give an introduction to the lower level Python Application
+Program Interfaces (APIs), and see how we use them to build our graph.
+Then we will build (effectively) the same graph in Keras, to see just
+how simple solving a machine learning problem can be.
+
+
+To install tensorflow on Unix/Linux systems, use pip as
-
+and/or if you use anaconda, just write (or install from the graphical user interface)
+
-class NeuralNetworkTensorflow:
- def __init__(
- self,
- X_train,
- Y_train,
- X_test,
- Y_test,
- n_neurons_layer1=100,
- n_neurons_layer2=50,
- n_categories=2,
- epochs=10,
- batch_size=100,
- eta=0.1,
- lmbd=0.0,
- ):
-
- # keep track of number of steps
- self.global_step = tf.Variable(0, dtype=tf.int32, trainable=False, name='global_step')
-
- self.X_train = X_train
- self.Y_train = Y_train
- self.X_test = X_test
- self.Y_test = Y_test
-
- self.n_inputs = X_train.shape[0]
- self.n_features = X_train.shape[1]
- self.n_neurons_layer1 = n_neurons_layer1
- self.n_neurons_layer2 = n_neurons_layer2
- self.n_categories = n_categories
-
- self.epochs = epochs
- self.batch_size = batch_size
- self.iterations = self.n_inputs // self.batch_size
- self.eta = eta
- self.lmbd = lmbd
-
- # build network piece by piece
- # name scopes (with) are used to enforce creation of new variables
- # https://www.tensorflow.org/guide/variables
- self.create_placeholders()
- self.create_DNN()
- self.create_loss()
- self.create_optimiser()
- self.create_accuracy()
-
- def create_placeholders(self):
- # placeholders are fine here, but "Datasets" are the preferred method
- # of streaming data into a model
- with tf.name_scope('data'):
- self.X = tf.placeholder(tf.float32, shape=(None, self.n_features), name='X_data')
- self.Y = tf.placeholder(tf.float32, shape=(None, self.n_categories), name='Y_data')
-
- def create_DNN(self):
- with tf.name_scope('DNN'):
- # the weights are stored to calculate regularization loss later
-
- # Fully connected layer 1
- self.W_fc1 = self.weight_variable([self.n_features, self.n_neurons_layer1], name='fc1', dtype=tf.float32)
- b_fc1 = self.bias_variable([self.n_neurons_layer1], name='fc1', dtype=tf.float32)
- a_fc1 = tf.nn.sigmoid(tf.matmul(self.X, self.W_fc1) + b_fc1)
-
- # Fully connected layer 2
- self.W_fc2 = self.weight_variable([self.n_neurons_layer1, self.n_neurons_layer2], name='fc2', dtype=tf.float32)
- b_fc2 = self.bias_variable([self.n_neurons_layer2], name='fc2', dtype=tf.float32)
- a_fc2 = tf.nn.sigmoid(tf.matmul(a_fc1, self.W_fc2) + b_fc2)
-
- # Output layer
- self.W_out = self.weight_variable([self.n_neurons_layer2, self.n_categories], name='out', dtype=tf.float32)
- b_out = self.bias_variable([self.n_categories], name='out', dtype=tf.float32)
- self.z_out = tf.matmul(a_fc2, self.W_out) + b_out
-
- def create_loss(self):
- with tf.name_scope('loss'):
- softmax_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=self.Y, logits=self.z_out))
-
- regularizer_loss_fc1 = tf.nn.l2_loss(self.W_fc1)
- regularizer_loss_fc2 = tf.nn.l2_loss(self.W_fc2)
- regularizer_loss_out = tf.nn.l2_loss(self.W_out)
- regularizer_loss = self.lmbd*(regularizer_loss_fc1 + regularizer_loss_fc2 + regularizer_loss_out)
-
- self.loss = softmax_loss + regularizer_loss
-
- def create_accuracy(self):
- with tf.name_scope('accuracy'):
- probabilities = tf.nn.softmax(self.z_out)
- predictions = tf.argmax(probabilities, axis=1)
- labels = tf.argmax(self.Y, axis=1)
-
- correct_predictions = tf.equal(predictions, labels)
- correct_predictions = tf.cast(correct_predictions, tf.float32)
- self.accuracy = tf.reduce_mean(correct_predictions)
-
- def create_optimiser(self):
- with tf.name_scope('optimizer'):
- self.optimizer = tf.train.GradientDescentOptimizer(learning_rate=self.eta).minimize(self.loss, global_step=self.global_step)
-
- def weight_variable(self, shape, name='', dtype=tf.float32):
- initial = tf.truncated_normal(shape, stddev=0.1)
- return tf.Variable(initial, name=name, dtype=dtype)
-
- def bias_variable(self, shape, name='', dtype=tf.float32):
- initial = tf.constant(0.1, shape=shape)
- return tf.Variable(initial, name=name, dtype=dtype)
-
- def fit(self):
- data_indices = np.arange(self.n_inputs)
-
- with tf.Session() as sess:
- sess.run(tf.global_variables_initializer())
- for i in range(self.epochs):
- for j in range(self.iterations):
- chosen_datapoints = np.random.choice(data_indices, size=self.batch_size, replace=False)
- batch_X, batch_Y = self.X_train[chosen_datapoints], self.Y_train[chosen_datapoints]
-
- sess.run([DNN.loss, DNN.optimizer],
- feed_dict={DNN.X: batch_X,
- DNN.Y: batch_Y})
- accuracy = sess.run(DNN.accuracy,
- feed_dict={DNN.X: batch_X,
- DNN.Y: batch_Y})
- step = sess.run(DNN.global_step)
-
- self.train_loss, self.train_accuracy = sess.run([DNN.loss, DNN.accuracy],
- feed_dict={DNN.X: self.X_train,
- DNN.Y: self.Y_train})
-
- self.test_loss, self.test_accuracy = sess.run([DNN.loss, DNN.accuracy],
- feed_dict={DNN.X: self.X_test,
- DNN.Y: self.Y_test})
+
+
@@ -391,6 +292,8 @@ MathJax.Hub.Config({
-
-
-
+# ensure the same random numbers appear every time
+np.random.seed(0)
-
-
-
@@ -325,6 +312,8 @@ writer.add_graph(tf54
-Keras is a high level neural network
-that supports Tensorflow, CTNK and Theano as backends.
-If you have Tensorflow installed Keras is available through the tf.keras module.
-If you have Anaconda installed you may run the following command
-
-
-
-
-Alternatively, if you have Tensorflow or one of the other supported backends install you may use the pip package manager:
+
-
-or look up the instructions here.
+
-
-
-
-
-
-
-
-
-
-
-
@@ -274,7 +275,7 @@ MathJax.Hub.Config({
@@ -387,7 +387,7 @@ $$
We have that \( \mathrm{W}_l \) is an \( N_{l-1} \times N_l \) matrix, while \( \hat{b}_l \) and \( \hat{y}_l \) are \( N_l \times 1 \) column vectors.
With this notation, the sum becomes a matrix-vector multiplication, and we can write
-the equation for the activations of hidden layer 2 as
+the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as
-For an MLP there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
+For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
Hereafter we will call the various entities of a layer for nodes.
There are also no connections within a single layer.
@@ -877,11 +877,10 @@ $$
-With the activation function \( \hat{z}^l \) we can in turn define the
+With the activation values \( \hat{z}^l \) we can in turn define the
output of layer \( l \) as \( \hat{a}^l = f(\hat{z}^l) \) where \( f \) is our
activation function. In the examples here we will use the sigmoid
-function discussed in our logistic regression lectures and here as
-well. We will also use the same activation function \( f \) for all layers
+function discussed in our logistic regression lectures. We will also use the same activation function \( f \) for all layers
and their nodes. It means we have
This is an important expression. The second term on the right handside
-measures how fast the cost is changing as a function of the $j$th
+measures how fast the cost function is changing as a function of the $j$th
output activation. If, for example, the cost function doesn't depend
much on a particular output node \( j \), then \( \delta_j^L \) will be small,
which is what we would expect. The first term on the right, measures
@@ -991,9 +990,9 @@ Notice that everything in the above equations is easily computed. In
particular, we compute \( z_j^L \) while computing the behaviour of the
network, and it is only a small additional overhead to compute
\( f'(z^L_j) \). The exact form of the derivative with respect to the
-outpuwill, of course, depend on the form of the cost function.
+output depends on the form of the cost function.
However, provided the cost function is known there should be little
-trouble computing
+trouble in calculating
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
+
+
Another interesting feature is that is when the activation function,
-represented by the sigmoid function here, is rather flat when towards
+represented by the sigmoid function here, is rather flat when we move towards
its end values \( 0 \) and \( 1 \) (see the above Python codes). In these
cases, the derivatives of the activation function will also be close
to zero, meaning again that the gradients will be small and the
@@ -1205,7 +1206,7 @@ $$
The parameter \( \eta \) is the learning parameter discussed in connection with the gradient descent methods.
-Here it is convenient to use stochastic radient descent with mini-batches with an outer loop that steps through multiple epochs of training.
+Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.
@@ -1230,11 +1231,11 @@ calculate.
For an input \( \boldsymbol{a} \) from the hidden layer, the probability that the input \( \boldsymbol{x} \)
-is in class 0 or 1 is just:
-
+is in class 0 or 1 is just. We let \( \theta \) represent the unknown weights and biases to be adjusted by our equations). The variable \( x \)
+represents our activation values \( z \). We have
@@ -1610,7 +1611,7 @@ Adding a bias value to the weighted sum of inputs allows the neural network to r
of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron \( j \), \( b_j \):
@@ -1642,32 +1643,32 @@ output_bias = np.zeros(n_categories) + 0.01
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
-For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer:
+For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer \( l \):
this is then passed through our activation function
We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron \( j \) in the output layer:
Finally we calculate the output of neuron \( j \) in the output layer using the softmax function:
@@ -1699,27 +1700,27 @@ meaning the same bias (1D array with size equal number of hidden neurons) is add
This is then passed through the activation:
This is fed to the output layer:
Finally we receive our output values for each image and each category by passing it through the softmax function:
-
The gradient for the output weights is calculated as
-where \( A = (n_{inputs}, n_{hidden}) \). This simply means that we are summing up the gradients for each input.
+where \( \hat{a} = (n_{inputs}, n_{hidden}) \). This simply means that we are summing up the gradients for each input.
Since we are going backwards we have to transpose the activation matrix.
The gradient with respect to the output bias is then
The error in the hidden layer is
-where \( f'(A_{h}) \) is the derivative of the activation in the hidden layer. The matrix products mean
+where \( f'(a_{h}) \) is the derivative of the activation in the hidden layer. The matrix products mean
that we are summing up the products for each neuron in the output layer. The symbol \( \circ \) denotes
the Hadamard product, meaning element-wise multiplication.
@@ -1929,11 +1930,11 @@ the Hadamard product, meaning element-wise multiplication.
This again gives us the gradients in the hidden layer:
@@ -2187,22 +2188,18 @@ test_predict = dnn.predict(X_test)
We now perform a grid search to find the optimal hyperparameters for the network.
-Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \( 98 \% \) (\( 2 \% \) error rate).
+Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \( 98\% \) (\( 2\% \) error rate).
-
-
-
-
-scikit-learn is a machine learning library for Python. It focuses more on traditional machine learning methods, such as regression, clustering, decision trees, etc. As such, it has only two types of neural networks: Multi Layer Perceptron outputting continuous values, MPLRegressor, and Multi Layer Perceptron outputting labels, MLPClassifier. We will see how simple it is to use these classes.
+scikit-learn focuses more
+on traditional machine learning methods, such as regression,
+clustering, decision trees, etc. As such, it has only two types of
+neural networks: Multi Layer Perceptron outputting continuous values,
+MPLRegressor, and Multi Layer Perceptron outputting labels,
+MLPClassifier. We will see how simple it is to use these classes.
-scikit-learn implements a few improvements from our neural network, such as early stopping, a varying learning rate, different optimization methods, etc. We would therefore expect a better performance overall.
+scikit-learn implements a few improvements from our neural network,
+such as early stopping, a varying learning rate, different
+optimization methods, etc. We would therefore expect a better
+performance overall.
@@ -2334,7 +2348,7 @@ plt.show()
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
@@ -2349,7 +2363,7 @@ NumPy arrays.
Tensorflow is an open source library machine learning library
@@ -2395,7 +2409,7 @@ and/or if you use anaconda, just write (or install from the graphical use
@@ -2462,7 +2476,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
@@ -2617,7 +2631,6 @@ batch_size = 100
n_neurons_layer1 = 100
n_neurons_layer2 = 50
n_categories = 10
-
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
@@ -2688,7 +2700,7 @@ writer.add_graph(tf.get_default_graph())
Keras is a high level neural network
diff --git a/doc/pub/NeuralNet/html/NeuralNet-solarized.html b/doc/pub/NeuralNet/html/NeuralNet-solarized.html
index 34459708f..ce4283891 100644
--- a/doc/pub/NeuralNet/html/NeuralNet-solarized.html
+++ b/doc/pub/NeuralNet/html/NeuralNet-solarized.html
@@ -130,20 +130,19 @@ div { text-align: justify; text-justify: inter-word; }
('Improving performance', 2, None, '___sec44'),
('Full object-oriented implementation', 2, None, '___sec45'),
('Evaluate model performance on test data', 2, None, '___sec46'),
- ('Adjust hyperparameters (if necessary, network architecture',
- 2,
- None,
- '___sec47'),
- ('scikit-learn implementation', 2, None, '___sec48'),
+ ('Adjust hyperparameters', 2, None, '___sec47'),
+ ('Visualization', 2, None, '___sec48'),
+ ('scikit-learn implementation', 2, None, '___sec49'),
+ ('Visualization', 2, None, '___sec50'),
('Building neural networks in Tensorflow and Keras',
2,
None,
- '___sec49'),
- ('Tensorflow', 2, None, '___sec50'),
- ('Collect and pre-process data', 2, None, '___sec51'),
- ('Using TensorFlow backend', 2, None, '___sec52'),
- ('Optimizing and using gradient descent', 2, None, '___sec53'),
- ('Using Keras', 2, None, '___sec54')]}
+ '___sec51'),
+ ('Tensorflow', 2, None, '___sec52'),
+ ('Collect and pre-process data', 2, None, '___sec53'),
+ ('Using TensorFlow backend', 2, None, '___sec54'),
+ ('Optimizing and using gradient descent', 2, None, '___sec55'),
+ ('Using Keras', 2, None, '___sec56')]}
end of tocinfo -->
-
We have that \( \mathrm{W}_l \) is an \( N_{l-1} \times N_l \) matrix, while \( \hat{b}_l \) and \( \hat{y}_l \) are \( N_l \times 1 \) column vectors.
With this notation, the sum becomes a matrix-vector multiplication, and we can write
-the equation for the activations of hidden layer 2 as
+the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as
$$
\begin{equation}
\hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) =
@@ -753,7 +752,7 @@ As a convention it is normal to call a network with one layer of input units,
units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.
-For an MLP there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
+For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
Hereafter we will call the various entities of a layer for nodes.
There are also no connections within a single layer.
@@ -863,11 +862,10 @@ $$
$$
-With the activation function \( \hat{z}^l \) we can in turn define the
+With the activation values \( \hat{z}^l \) we can in turn define the
output of layer \( l \) as \( \hat{a}^l = f(\hat{z}^l) \) where \( f \) is our
activation function. In the examples here we will use the sigmoid
-function discussed in our logistic regression lectures and here as
-well. We will also use the same activation function \( f \) for all layers
+function discussed in our logistic regression lectures. We will also use the same activation function \( f \) for all layers
and their nodes. It means we have
$$
@@ -945,7 +943,7 @@ $$
This is an important expression. The second term on the right handside
-measures how fast the cost is changing as a function of the $j$th
+measures how fast the cost function is changing as a function of the $j$th
output activation. If, for example, the cost function doesn't depend
much on a particular output node \( j \), then \( \delta_j^L \) will be small,
which is what we would expect. The first term on the right, measures
@@ -957,9 +955,9 @@ Notice that everything in the above equations is easily computed. In
particular, we compute \( z_j^L \) while computing the behaviour of the
network, and it is only a small additional overhead to compute
\( f'(z^L_j) \). The exact form of the derivative with respect to the
-outpuwill, of course, depend on the form of the cost function.
+output depends on the form of the cost function.
However, provided the cost function is known there should be little
-trouble computing
+trouble in calculating
$$
\frac{\partial {\cal C}}{\partial (a_j^L)}
@@ -995,6 +993,8 @@ That is, the error \( \delta_j^L \) is exactly equal to the rate of change of th
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
+
+
@@ -1035,7 +1035,7 @@ descent. In this case we say the system learns slowly.
Another interesting feature is that is when the activation function,
-represented by the sigmoid function here, is rather flat when towards
+represented by the sigmoid function here, is rather flat when we move towards
its end values \( 0 \) and \( 1 \) (see the above Python codes). In these
cases, the derivatives of the activation function will also be close
to zero, meaning again that the gradients will be small and the
@@ -1148,7 +1148,7 @@ $$
The parameter \( \eta \) is the learning parameter discussed in connection with the gradient descent methods.
-Here it is convenient to use stochastic radient descent with mini-batches with an outer loop that steps through multiple epochs of training.
+Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.
@@ -1173,10 +1173,10 @@ calculate.
For an input \( \boldsymbol{a} \) from the hidden layer, the probability that the input \( \boldsymbol{x} \)
-is in class 0 or 1 is just:
-
+is in class 0 or 1 is just. We let \( \theta \) represent the unknown weights and biases to be adjusted by our equations). The variable \( x \)
+represents our activation values \( z \). We have
$$
-P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{a}^T \boldsymbol{w}_{out})} ,
+P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{x}} ,
$$
and
@@ -1494,7 +1494,7 @@ i.e. each neuron \( j \) outputs the probability of being in class \( j \) given
The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1.
The exponent is just the weighted sum of inputs as before:
-$$ z_j = \sum_{i=1}^n w_ {ij} a_i = \boldsymbol{a}^T \boldsymbol{w}_j .$$
+$$ z_j = \sum_{i=1}^n w_ {ij} a_i+b_j.$$
Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500
@@ -1513,7 +1513,7 @@ or normal distribution. Setting all weights to zero means all neurons give the s
Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range
of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron \( j \), \( b_j \):
-$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j = \boldsymbol{a}^T \boldsymbol{w}_j + b_j .$$
+$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j.$$
The bias weights \( \boldsymbol{b} \) are often initialized to zero, but a small value like \( 0.01 \) ensures all neurons have some output which can be backpropagated in the first training cycle.
@@ -1543,25 +1543,25 @@ output_bias = np.zeros(n_categories) + 0.01
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
-For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer:
+For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer \( l \):
-$$ z_{j}^{h} = \sum_{i=1}^{F} w_{ij}^{h} x_i + b_{j}^{h} = \boldsymbol{x}^T \boldsymbol{w}_{j}^{h} + b_{j}^{h} ,$$
+$$ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$
this is then passed through our activation function
-$$ a_{j}^{h} = f(z_{j}^{h}) .$$
+$$ a_{j}^{l} = f(z_{j}^{l}) .$$
We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron \( j \) in the output layer:
-$$ z_{j}^{o} = \sum_{i=1}^{H} w_{ij}^{o} a_{i}^{h} + b_{j}^{o} = (\boldsymbol{a}^{h})^T \boldsymbol{w}_{j}^{o} + b_{j}^{o} .$$
+$$ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$
Finally we calculate the output of neuron \( j \) in the output layer using the softmax function:
-$$ a_{j}^{o} = \frac{\exp{(z_j^{o})}}
-{\sum_{c=0}^{C-1} \exp{(z_c^{o})}} .$$
+$$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
+{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .$$
@@ -1581,28 +1581,28 @@ and obtain a matrix that holds the weighted sum of inputs to the hidden layer
for each input image and each hidden neuron.
We also add the bias to obtain a matrix of weighted sums to the hidden layer \( Z^{h} \):
-$$ Z^{h} = X W^{h} + B^{h} ,$$
+$$ \hat{z}^{l} = \hat{X} \hat{W}^{l} + \hat{b}^{l} ,$$
meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image.
This is then passed through the activation:
-$$ A^{h} = f(Z^h) .$$
+$$ \hat{a}^{l} = f(\hat{z}^l) .$$
This is fed to the output layer:
-$$ Z^{o} = A^{h} W^{o} + B^{o} .$$
+$$ \hat{z}^{L} = \hat{a}^{L} \hat{W}^{L} + \hat{b}^{L} .$$
Finally we receive our output values for each image and each category by passing it through the softmax function:
-$$ output = softmax (Z^{o}) = (n_{inputs}, n_{categories}) .$$
+$$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
-
The gradient for the output weights is calculated as
-$$ \nabla W_{o} = A^T \Delta_o = (n_{hidden}, n_{categories}) ,$$
+$$ \nabla W_{L} = \hat{a}^T \delta_L = (n_{hidden}, n_{categories}) ,$$
-where \( A = (n_{inputs}, n_{hidden}) \). This simply means that we are summing up the gradients for each input.
+where \( \hat{a} = (n_{inputs}, n_{hidden}) \). This simply means that we are summing up the gradients for each input.
Since we are going backwards we have to transpose the activation matrix.
The gradient with respect to the output bias is then
-$$ \nabla B_{o} = \sum_{i=1}^{n_{inputs}} \Delta_o = (n_{categories}) .$$
+$$ \nabla \hat{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .$$
The error in the hidden layer is
-$$ \Delta_h = \Delta_o W_{o}^T \circ f'(Z_{h}) = \Delta_o W_{o}^T \circ A_{h} \circ (1 - A_{h}) = (n_{inputs}, n_{hidden}) ,$$
+$$ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$
-where \( f'(A_{h}) \) is the derivative of the activation in the hidden layer. The matrix products mean
+where \( f'(a_{h}) \) is the derivative of the activation in the hidden layer. The matrix products mean
that we are summing up the products for each neuron in the output layer. The symbol \( \circ \) denotes
the Hadamard product, meaning element-wise multiplication.
This again gives us the gradients in the hidden layer:
-$$ \nabla W_{h} = X^T \Delta_h = (n_{features}, n_{hidden}) ,$$
+$$ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,$$
-$$ \nabla B_{h} = \sum_{i=1}^{n_{inputs}} \Delta_h = (n_{hidden}) .$$
+$$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
@@ -2038,22 +2038,18 @@ test_predict = dnn.predict(X_test)
We now perform a grid search to find the optimal hyperparameters for the network.
-Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \( 98 \% \) (\( 2 \% \) error rate).
+Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \( 98\% \) (\( 2\% \) error rate).
-
-
-
+
-
-scikit-learn is a machine learning library for Python. It focuses more on traditional machine learning methods, such as regression, clustering, decision trees, etc. As such, it has only two types of neural networks: Multi Layer Perceptron outputting continuous values, MPLRegressor, and Multi Layer Perceptron outputting labels, MLPClassifier. We will see how simple it is to use these classes.
+scikit-learn focuses more
+on traditional machine learning methods, such as regression,
+clustering, decision trees, etc. As such, it has only two types of
+neural networks: Multi Layer Perceptron outputting continuous values,
+MPLRegressor, and Multi Layer Perceptron outputting labels,
+MLPClassifier. We will see how simple it is to use these classes.
-scikit-learn implements a few improvements from our neural network, such as early stopping, a varying learning rate, different optimization methods, etc. We would therefore expect a better performance overall.
+scikit-learn implements a few improvements from our neural network,
+such as early stopping, a varying learning rate, different
+optimization methods, etc. We would therefore expect a better
+performance overall.
+
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
@@ -2198,7 +2209,7 @@ NumPy arrays.
Tensorflow is an open source library machine learning library
@@ -2243,7 +2254,7 @@ and/or if you use anaconda, just write (or install from the graphical use
@@ -2309,7 +2320,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
@@ -2464,7 +2475,6 @@ batch_size = 100
n_neurons_layer1 = 100
n_neurons_layer2 = 50
n_categories = 10
-
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
@@ -2534,7 +2543,7 @@ writer.add_graph(tf.get_default_graph())
Keras is a high level neural network
diff --git a/doc/pub/NeuralNet/html/NeuralNet.html b/doc/pub/NeuralNet/html/NeuralNet.html
index cfb13c6ce..5a2aaa6cc 100644
--- a/doc/pub/NeuralNet/html/NeuralNet.html
+++ b/doc/pub/NeuralNet/html/NeuralNet.html
@@ -135,20 +135,19 @@ div { text-align: justify; text-justify: inter-word; }
('Improving performance', 2, None, '___sec44'),
('Full object-oriented implementation', 2, None, '___sec45'),
('Evaluate model performance on test data', 2, None, '___sec46'),
- ('Adjust hyperparameters (if necessary, network architecture',
- 2,
- None,
- '___sec47'),
- ('scikit-learn implementation', 2, None, '___sec48'),
+ ('Adjust hyperparameters', 2, None, '___sec47'),
+ ('Visualization', 2, None, '___sec48'),
+ ('scikit-learn implementation', 2, None, '___sec49'),
+ ('Visualization', 2, None, '___sec50'),
('Building neural networks in Tensorflow and Keras',
2,
None,
- '___sec49'),
- ('Tensorflow', 2, None, '___sec50'),
- ('Collect and pre-process data', 2, None, '___sec51'),
- ('Using TensorFlow backend', 2, None, '___sec52'),
- ('Optimizing and using gradient descent', 2, None, '___sec53'),
- ('Using Keras', 2, None, '___sec54')]}
+ '___sec51'),
+ ('Tensorflow', 2, None, '___sec52'),
+ ('Collect and pre-process data', 2, None, '___sec53'),
+ ('Using TensorFlow backend', 2, None, '___sec54'),
+ ('Optimizing and using gradient descent', 2, None, '___sec55'),
+ ('Using Keras', 2, None, '___sec56')]}
end of tocinfo -->
-
We have that \( \mathrm{W}_l \) is an \( N_{l-1} \times N_l \) matrix, while \( \hat{b}_l \) and \( \hat{y}_l \) are \( N_l \times 1 \) column vectors.
With this notation, the sum becomes a matrix-vector multiplication, and we can write
-the equation for the activations of hidden layer 2 as
+the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as
$$
\begin{equation}
\hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) =
@@ -758,7 +757,7 @@ As a convention it is normal to call a network with one layer of input units,
units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.
-For an MLP there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
+For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
Hereafter we will call the various entities of a layer for nodes.
There are also no connections within a single layer.
@@ -868,11 +867,10 @@ $$
$$
-With the activation function \( \hat{z}^l \) we can in turn define the
+With the activation values \( \hat{z}^l \) we can in turn define the
output of layer \( l \) as \( \hat{a}^l = f(\hat{z}^l) \) where \( f \) is our
activation function. In the examples here we will use the sigmoid
-function discussed in our logistic regression lectures and here as
-well. We will also use the same activation function \( f \) for all layers
+function discussed in our logistic regression lectures. We will also use the same activation function \( f \) for all layers
and their nodes. It means we have
$$
@@ -950,7 +948,7 @@ $$
This is an important expression. The second term on the right handside
-measures how fast the cost is changing as a function of the $j$th
+measures how fast the cost function is changing as a function of the $j$th
output activation. If, for example, the cost function doesn't depend
much on a particular output node \( j \), then \( \delta_j^L \) will be small,
which is what we would expect. The first term on the right, measures
@@ -962,9 +960,9 @@ Notice that everything in the above equations is easily computed. In
particular, we compute \( z_j^L \) while computing the behaviour of the
network, and it is only a small additional overhead to compute
\( f'(z^L_j) \). The exact form of the derivative with respect to the
-outpuwill, of course, depend on the form of the cost function.
+output depends on the form of the cost function.
However, provided the cost function is known there should be little
-trouble computing
+trouble in calculating
$$
\frac{\partial {\cal C}}{\partial (a_j^L)}
@@ -1000,6 +998,8 @@ That is, the error \( \delta_j^L \) is exactly equal to the rate of change of th
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
+
+
@@ -1040,7 +1040,7 @@ descent. In this case we say the system learns slowly.
Another interesting feature is that is when the activation function,
-represented by the sigmoid function here, is rather flat when towards
+represented by the sigmoid function here, is rather flat when we move towards
its end values \( 0 \) and \( 1 \) (see the above Python codes). In these
cases, the derivatives of the activation function will also be close
to zero, meaning again that the gradients will be small and the
@@ -1153,7 +1153,7 @@ $$
The parameter \( \eta \) is the learning parameter discussed in connection with the gradient descent methods.
-Here it is convenient to use stochastic radient descent with mini-batches with an outer loop that steps through multiple epochs of training.
+Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.
@@ -1178,10 +1178,10 @@ calculate.
For an input \( \boldsymbol{a} \) from the hidden layer, the probability that the input \( \boldsymbol{x} \)
-is in class 0 or 1 is just:
-
+is in class 0 or 1 is just. We let \( \theta \) represent the unknown weights and biases to be adjusted by our equations). The variable \( x \)
+represents our activation values \( z \). We have
$$
-P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{a}^T \boldsymbol{w}_{out})} ,
+P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{x}} ,
$$
and
@@ -1499,7 +1499,7 @@ i.e. each neuron \( j \) outputs the probability of being in class \( j \) given
The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1.
The exponent is just the weighted sum of inputs as before:
-$$ z_j = \sum_{i=1}^n w_ {ij} a_i = \boldsymbol{a}^T \boldsymbol{w}_j .$$
+$$ z_j = \sum_{i=1}^n w_ {ij} a_i+b_j.$$
Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500
@@ -1518,7 +1518,7 @@ or normal distribution. Setting all weights to zero means all neurons give the s
Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range
of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron \( j \), \( b_j \):
-$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j = \boldsymbol{a}^T \boldsymbol{w}_j + b_j .$$
+$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j.$$
The bias weights \( \boldsymbol{b} \) are often initialized to zero, but a small value like \( 0.01 \) ensures all neurons have some output which can be backpropagated in the first training cycle.
@@ -1548,25 +1548,25 @@ output_bias = np
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
-For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer:
+For each input image we calculate a weighted sum of input features (pixel values) to each neuron \( j \) in the hidden layer \( l \):
-$$ z_{j}^{h} = \sum_{i=1}^{F} w_{ij}^{h} x_i + b_{j}^{h} = \boldsymbol{x}^T \boldsymbol{w}_{j}^{h} + b_{j}^{h} ,$$
+$$ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$
this is then passed through our activation function
-$$ a_{j}^{h} = f(z_{j}^{h}) .$$
+$$ a_{j}^{l} = f(z_{j}^{l}) .$$
We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron \( j \) in the output layer:
-$$ z_{j}^{o} = \sum_{i=1}^{H} w_{ij}^{o} a_{i}^{h} + b_{j}^{o} = (\boldsymbol{a}^{h})^T \boldsymbol{w}_{j}^{o} + b_{j}^{o} .$$
+$$ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$
Finally we calculate the output of neuron \( j \) in the output layer using the softmax function:
-$$ a_{j}^{o} = \frac{\exp{(z_j^{o})}}
-{\sum_{c=0}^{C-1} \exp{(z_c^{o})}} .$$
+$$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
+{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .$$
@@ -1586,28 +1586,28 @@ and obtain a matrix that holds the weighted sum of inputs to the hidden layer
for each input image and each hidden neuron.
We also add the bias to obtain a matrix of weighted sums to the hidden layer \( Z^{h} \):
-$$ Z^{h} = X W^{h} + B^{h} ,$$
+$$ \hat{z}^{l} = \hat{X} \hat{W}^{l} + \hat{b}^{l} ,$$
meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image.
This is then passed through the activation:
-$$ A^{h} = f(Z^h) .$$
+$$ \hat{a}^{l} = f(\hat{z}^l) .$$
This is fed to the output layer:
-$$ Z^{o} = A^{h} W^{o} + B^{o} .$$
+$$ \hat{z}^{L} = \hat{a}^{L} \hat{W}^{L} + \hat{b}^{L} .$$
Finally we receive our output values for each image and each category by passing it through the softmax function:
-$$ output = softmax (Z^{o}) = (n_{inputs}, n_{categories}) .$$
+$$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
-
The gradient for the output weights is calculated as
-$$ \nabla W_{o} = A^T \Delta_o = (n_{hidden}, n_{categories}) ,$$
+$$ \nabla W_{L} = \hat{a}^T \delta_L = (n_{hidden}, n_{categories}) ,$$
-where \( A = (n_{inputs}, n_{hidden}) \). This simply means that we are summing up the gradients for each input.
+where \( \hat{a} = (n_{inputs}, n_{hidden}) \). This simply means that we are summing up the gradients for each input.
Since we are going backwards we have to transpose the activation matrix.
The gradient with respect to the output bias is then
-$$ \nabla B_{o} = \sum_{i=1}^{n_{inputs}} \Delta_o = (n_{categories}) .$$
+$$ \nabla \hat{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .$$
The error in the hidden layer is
-$$ \Delta_h = \Delta_o W_{o}^T \circ f'(Z_{h}) = \Delta_o W_{o}^T \circ A_{h} \circ (1 - A_{h}) = (n_{inputs}, n_{hidden}) ,$$
+$$ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$
-where \( f'(A_{h}) \) is the derivative of the activation in the hidden layer. The matrix products mean
+where \( f'(a_{h}) \) is the derivative of the activation in the hidden layer. The matrix products mean
that we are summing up the products for each neuron in the output layer. The symbol \( \circ \) denotes
the Hadamard product, meaning element-wise multiplication.
This again gives us the gradients in the hidden layer:
-$$ \nabla W_{h} = X^T \Delta_h = (n_{features}, n_{hidden}) ,$$
+$$ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,$$
-$$ \nabla B_{h} = \sum_{i=1}^{n_{inputs}} \Delta_h = (n_{hidden}) .$$
+$$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
@@ -2043,22 +2043,18 @@ test_predict = dnnAdjust hyperparameters (if necessary, network architecture
+
We now perform a grid search to find the optimal hyperparameters for the network.
-Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \( 98 \% \) (\( 2 \% \) error rate).
+Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \( 98\% \) (\( 2\% \) error rate).
-
-
-
+
-
-scikit-learn is a machine learning library for Python. It focuses more on traditional machine learning methods, such as regression, clustering, decision trees, etc. As such, it has only two types of neural networks: Multi Layer Perceptron outputting continuous values, MPLRegressor, and Multi Layer Perceptron outputting labels, MLPClassifier. We will see how simple it is to use these classes.
+scikit-learn focuses more
+on traditional machine learning methods, such as regression,
+clustering, decision trees, etc. As such, it has only two types of
+neural networks: Multi Layer Perceptron outputting continuous values,
+MPLRegressor, and Multi Layer Perceptron outputting labels,
+MLPClassifier. We will see how simple it is to use these classes.
-scikit-learn implements a few improvements from our neural network, such as early stopping, a varying learning rate, different optimization methods, etc. We would therefore expect a better performance overall.
+scikit-learn implements a few improvements from our neural network,
+such as early stopping, a varying learning rate, different
+optimization methods, etc. We would therefore expect a better
+performance overall.
+
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
@@ -2203,7 +2214,7 @@ NumPy arrays.
Tensorflow is an open source library machine learning library
@@ -2248,7 +2259,7 @@ and/or if you use anaconda, just write (or install from the graphical use
@@ -2314,7 +2325,7 @@ X_train, X_test, Y_train, Y_test = train_tes
@@ -2469,7 +2480,6 @@ batch_size = 10
n_neurons_layer1 = 100
n_neurons_layer2 = 50
n_categories = 10
-
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
@@ -2539,7 +2548,7 @@ writer.add_graph(tfUsing Keras
+
Keras is a high level neural network
diff --git a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
index fd6d5ee07..6d7b7115e 100644
--- a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
+++ b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Oct 2, 2018**\n",
+ "Date: **Oct 4, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -222,7 +222,7 @@
"metadata": {},
"source": [
"This function receives $x_i$ as inputs.\n",
- "Here the activation $z=\\sum_{i=1}^n w_ix_i$. \n",
+ "Here the activation $z=(\\sum_{i=1}^n w_ix_i+b_i)$. \n",
"In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of\n",
"the neurons in the preceding layer. Furthermore, an MLP is\n",
"fully-connected, which means that each neuron receives a weighted sum\n",
@@ -460,7 +460,7 @@
"\n",
"We have that $\\mathrm{W}_l$ is an $N_{l-1} \\times N_l$ matrix, while $\\hat{b}_l$ and $\\hat{y}_l$ are $N_l \\times 1$ column vectors. \n",
"With this notation, the sum becomes a matrix-vector multiplication, and we can write\n",
- "the equation for the activations of hidden layer 2 as"
+ "the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as"
]
},
{
@@ -693,7 +693,7 @@
"As a convention it is normal to call a network with one layer of input units, one layer of hidden\n",
"units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.\n",
"\n",
- "For an MLP there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.\n",
+ "For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.\n",
"Hereafter we will call the various entities of a layer for nodes.\n",
"There are also no connections within a single layer.\n",
"\n",
@@ -815,11 +815,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "With the activation function $\\hat{z}^l$ we can in turn define the\n",
+ "With the activation values $\\hat{z}^l$ we can in turn define the\n",
"output of layer $l$ as $\\hat{a}^l = f(\\hat{z}^l)$ where $f$ is our\n",
"activation function. In the examples here we will use the sigmoid\n",
- "function discussed in our logistic regression lectures and here as\n",
- "well. We will also use the same activation function $f$ for all layers\n",
+ "function discussed in our logistic regression lectures. We will also use the same activation function $f$ for all layers\n",
"and their nodes. It means we have"
]
},
@@ -989,7 +988,7 @@
"metadata": {},
"source": [
"This is an important expression. The second term on the right handside\n",
- "measures how fast the cost is changing as a function of the $j$th\n",
+ "measures how fast the cost function is changing as a function of the $j$th\n",
"output activation. If, for example, the cost function doesn't depend\n",
"much on a particular output node $j$, then $\\delta_j^L$ will be small,\n",
"which is what we would expect. The first term on the right, measures\n",
@@ -1000,9 +999,9 @@
"particular, we compute $z_j^L$ while computing the behaviour of the\n",
"network, and it is only a small additional overhead to compute\n",
"$f'(z^L_j)$. The exact form of the derivative with respect to the\n",
- "outpuwill, of course, depend on the form of the cost function.\n",
+ "output depends on the form of the cost function.\n",
"However, provided the cost function is known there should be little\n",
- "trouble computing"
+ "trouble in calculating"
]
},
{
@@ -1072,6 +1071,7 @@
"## Bringing it together\n",
"\n",
"We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are\n",
+ "\n",
"**The starting equations.**"
]
},
@@ -1146,7 +1146,7 @@
"descent. In this case we say the system learns slowly.\n",
"\n",
"Another interesting feature is that is when the activation function,\n",
- "represented by the sigmoid function here, is rather flat when towards\n",
+ "represented by the sigmoid function here, is rather flat when we move towards\n",
"its end values $0$ and $1$ (see the above Python codes). In these\n",
"cases, the derivatives of the activation function will also be close\n",
"to zero, meaning again that the gradients will be small and the\n",
@@ -1306,7 +1306,7 @@
"metadata": {},
"source": [
"The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n",
- "Here it is convenient to use stochastic radient descent with mini-batches with an outer loop that steps through multiple epochs of training.\n",
+ "Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.\n",
"\n",
"\n",
"\n",
@@ -1327,7 +1327,8 @@
"calculate.\n",
"\n",
"For an input $\\boldsymbol{a}$ from the hidden layer, the probability that the input $\\boldsymbol{x}$\n",
- "is in class 0 or 1 is just:"
+ "is in class 0 or 1 is just. We let $\\theta$ represent the unknown weights and biases to be adjusted by our equations). The variable $x$\n",
+ "represents our activation values $z$. We have"
]
},
{
@@ -1335,7 +1336,7 @@
"metadata": {},
"source": [
"$$\n",
- "P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp (- \\boldsymbol{a}^T \\boldsymbol{w}_{out})} ,\n",
+ "P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp (- \\boldsymbol{x}} ,\n",
"$$"
]
},
@@ -1690,7 +1691,7 @@
"The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1. \n",
"The exponent is just the weighted sum of inputs as before: \n",
"\n",
- "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i = \\boldsymbol{a}^T \\boldsymbol{w}_j .$$ \n",
+ "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i+b_j.$$ \n",
"\n",
"Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500\n",
"weights to the output layer.\n",
@@ -1704,7 +1705,7 @@
"Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range\n",
"of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron $j$, $b_j$: \n",
"\n",
- "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i + 1\\cdot b_j = \\boldsymbol{a}^T \\boldsymbol{w}_j + b_j .$$ \n",
+ "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i + 1\\cdot b_j.$$ \n",
"\n",
"The bias weights $\\boldsymbol{b}$ are often initialized to zero, but a small value like $0.01$ ensures all neurons have some output which can be backpropagated in the first training cycle."
]
@@ -1741,22 +1742,22 @@
"## Feed-forward pass\n",
"\n",
"Denote $F$ the number of features, $H$ the number of hidden neurons and $C$ the number of categories. \n",
- "For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer: \n",
+ "For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer $l$: \n",
"\n",
- "$$ z_{j}^{h} = \\sum_{i=1}^{F} w_{ij}^{h} x_i + b_{j}^{h} = \\boldsymbol{x}^T \\boldsymbol{w}_{j}^{h} + b_{j}^{h} ,$$\n",
+ "$$ z_{j}^{l} = \\sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$\n",
"\n",
"this is then passed through our activation function \n",
"\n",
- "$$ a_{j}^{h} = f(z_{j}^{h}) .$$ \n",
+ "$$ a_{j}^{l} = f(z_{j}^{l}) .$$ \n",
"\n",
"We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron $j$ in the output layer: \n",
"\n",
- "$$ z_{j}^{o} = \\sum_{i=1}^{H} w_{ij}^{o} a_{i}^{h} + b_{j}^{o} = (\\boldsymbol{a}^{h})^T \\boldsymbol{w}_{j}^{o} + b_{j}^{o} .$$ \n",
+ "$$ z_{j}^{L} = \\sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$ \n",
"\n",
"Finally we calculate the output of neuron $j$ in the output layer using the softmax function: \n",
"\n",
- "$$ a_{j}^{o} = \\frac{\\exp{(z_j^{o})}}\n",
- "{\\sum_{c=0}^{C-1} \\exp{(z_c^{o})}} .$$ \n",
+ "$$ a_{j}^{L} = \\frac{\\exp{(z_j^{L})}}\n",
+ "{\\sum_{c=0}^{C-1} \\exp{(z_c^{L})}} .$$ \n",
"\n",
"\n",
"## Matrix multiplication\n",
@@ -1772,20 +1773,20 @@
"for each input image and each hidden neuron. \n",
"We also add the bias to obtain a matrix of weighted sums to the hidden layer $Z^{h}$: \n",
"\n",
- "$$ Z^{h} = X W^{h} + B^{h} ,$$\n",
+ "$$ \\hat{z}^{l} = \\hat{X} \\hat{W}^{l} + \\hat{b}^{l} ,$$\n",
"\n",
"meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image. \n",
"This is then passed through the activation: \n",
"\n",
- "$$ A^{h} = f(Z^h) .$$ \n",
+ "$$ \\hat{a}^{l} = f(\\hat{z}^l) .$$ \n",
"\n",
"This is fed to the output layer: \n",
"\n",
- "$$ Z^{o} = A^{h} W^{o} + B^{o} .$$\n",
+ "$$ \\hat{z}^{L} = \\hat{a}^{L} \\hat{W}^{L} + \\hat{b}^{L} .$$\n",
"\n",
"Finally we receive our output values for each image and each category by passing it through the softmax function: \n",
"\n",
- "$$ output = softmax (Z^{o}) = (n_{inputs}, n_{categories}) .$$"
+ "$$ output = softmax (\\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$"
]
},
{
@@ -1796,7 +1797,7 @@
},
"outputs": [],
"source": [
- "# setup the feed-forward pass\n",
+ "# setup the feed-forward pass, subscript h = hidden layer\n",
"\n",
"def sigmoid(x):\n",
" return 1/(1 + np.exp(-x))\n",
@@ -1960,32 +1961,32 @@
"To more efficently train our network these equations are implemented using matrix operations. \n",
"The error in the output layer is calculated simply as \n",
"\n",
- "$$ \\Delta_o = \\hat{y} - y = (n_{inputs}, n_{categories}) .$$ \n",
+ "$$ \\delta_L = \\hat{y} - y = (n_{inputs}, n_{categories}) .$$ \n",
"\n",
"The gradient for the output weights is calculated as \n",
"\n",
- "$$ \\nabla W_{o} = A^T \\Delta_o = (n_{hidden}, n_{categories}) ,$$\n",
+ "$$ \\nabla W_{L} = \\hat{a}^T \\delta_L = (n_{hidden}, n_{categories}) ,$$\n",
"\n",
- "where $A = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input. \n",
+ "where $\\hat{a} = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input. \n",
"Since we are going backwards we have to transpose the activation matrix. \n",
"\n",
"The gradient with respect to the output bias is then \n",
"\n",
- "$$ \\nabla B_{o} = \\sum_{i=1}^{n_{inputs}} \\Delta_o = (n_{categories}) .$$ \n",
+ "$$ \\nabla \\hat{b}_{L} = \\sum_{i=1}^{n_{inputs}} \\delta_L = (n_{categories}) .$$ \n",
"\n",
"The error in the hidden layer is \n",
"\n",
- "$$ \\Delta_h = \\Delta_o W_{o}^T \\circ f'(Z_{h}) = \\Delta_o W_{o}^T \\circ A_{h} \\circ (1 - A_{h}) = (n_{inputs}, n_{hidden}) ,$$ \n",
+ "$$ \\Delta_h = \\delta_L W_{L}^T \\circ f'(z_{h}) = \\delta_L W_{L}^T \\circ a_{h} \\circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$ \n",
"\n",
- "where $f'(A_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean\n",
+ "where $f'(a_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean\n",
"that we are summing up the products for each neuron in the output layer. The symbol $\\circ$ denotes\n",
"the *Hadamard product*, meaning element-wise multiplication. \n",
"\n",
"This again gives us the gradients in the hidden layer: \n",
"\n",
- "$$ \\nabla W_{h} = X^T \\Delta_h = (n_{features}, n_{hidden}) ,$$ \n",
+ "$$ \\nabla W_{h} = X^T \\delta_h = (n_{features}, n_{hidden}) ,$$ \n",
"\n",
- "$$ \\nabla B_{h} = \\sum_{i=1}^{n_{inputs}} \\Delta_h = (n_{hidden}) .$$"
+ "$$ \\nabla b_{h} = \\sum_{i=1}^{n_{inputs}} \\delta_h = (n_{hidden}) .$$"
]
},
{
@@ -2247,70 +2248,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Adjust hyperparameters (if necessary, network architecture\n",
+ "## Adjust hyperparameters\n",
"\n",
"We now perform a grid search to find the optimal hyperparameters for the network. \n",
- "Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98 \\%$ ($2 \\%$ error rate)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "8\n",
- " \n",
- "<\n",
- "<\n",
- "<\n",
- "!\n",
- "!\n",
- "C\n",
- "O\n",
- "D\n",
- "E\n",
- "_\n",
- "B\n",
- "L\n",
- "O\n",
- "C\n",
- "K\n",
- " \n",
- " \n",
- "p\n",
- "y\n",
- "c\n",
- "o\n",
- "d"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "9\n",
- " \n",
- "<\n",
- "<\n",
- "<\n",
- "!\n",
- "!\n",
- "C\n",
- "O\n",
- "D\n",
- "E\n",
- "_\n",
- "B\n",
- "L\n",
- "O\n",
- "C\n",
- "K\n",
- " \n",
- " \n",
- "p\n",
- "y\n",
- "c\n",
- "o\n",
- "d"
+ "Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\\%$ ($2\\%$ error rate)."
]
},
{
@@ -2321,9 +2262,45 @@
},
"outputs": [],
"source": [
- "# optional\n",
+ "eta_vals = np.logspace(-5, 1, 7)\n",
+ "lmbd_vals = np.logspace(-5, 1, 7)\n",
+ "# store the models for later use\n",
+ "DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n",
+ "\n",
+ "# grid search\n",
+ "for i, eta in enumerate(eta_vals):\n",
+ " for j, lmbd in enumerate(lmbd_vals):\n",
+ " dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n",
+ " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n",
+ " dnn.train()\n",
+ " \n",
+ " DNN_numpy[i][j] = dnn\n",
+ " \n",
+ " test_predict = dnn.predict(X_test)\n",
+ " \n",
+ " print(\"Learning rate = \", eta)\n",
+ " print(\"Lambda = \", lmbd)\n",
+ " print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n",
+ " print()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Visualization"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
"# visual representation of grid search\n",
- "# uses seaborn heatmap, I believe you can also do this with matplotlib imshow\n",
+ "# uses seaborn heatmap, you can also do this with matplotlib imshow\n",
"import seaborn as sns\n",
"\n",
"sns.set()\n",
@@ -2363,45 +2340,55 @@
"source": [
"## scikit-learn implementation\n",
"\n",
- "scikit-learn is a machine learning library for Python. It focuses more on traditional machine learning methods, such as regression, clustering, decision trees, etc. As such, it has only two types of neural networks: Multi Layer Perceptron outputting continuous values, *MPLRegressor*, and Multi Layer Perceptron outputting labels, *MLPClassifier*. We will see how simple it is to use these classes. \n",
+ "**scikit-learn** focuses more\n",
+ "on traditional machine learning methods, such as regression,\n",
+ "clustering, decision trees, etc. As such, it has only two types of\n",
+ "neural networks: Multi Layer Perceptron outputting continuous values,\n",
+ "*MPLRegressor*, and Multi Layer Perceptron outputting labels,\n",
+ "*MLPClassifier*. We will see how simple it is to use these classes.\n",
"\n",
- "scikit-learn implements a few improvements from our neural network, such as early stopping, a varying learning rate, different optimization methods, etc. We would therefore expect a better performance overall."
+ "**scikit-learn** implements a few improvements from our neural network,\n",
+ "such as early stopping, a varying learning rate, different\n",
+ "optimization methods, etc. We would therefore expect a better\n",
+ "performance overall."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "from sklearn.neural_network import MLPClassifier\n",
+ "# store models for later use\n",
+ "DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n",
+ "\n",
+ "for i, eta in enumerate(eta_vals):\n",
+ " for j, lmbd in enumerate(lmbd_vals):\n",
+ " dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n",
+ " alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n",
+ " dnn.fit(X_train, Y_train)\n",
+ " \n",
+ " DNN_scikit[i][j] = dnn\n",
+ " \n",
+ " print(\"Learning rate = \", eta)\n",
+ " print(\"Lambda = \", lmbd)\n",
+ " print(\"Accuracy score on test set: \", dnn.score(X_test, Y_test))\n",
+ " print()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "1\n",
- "1\n",
- " \n",
- "<\n",
- "<\n",
- "<\n",
- "!\n",
- "!\n",
- "C\n",
- "O\n",
- "D\n",
- "E\n",
- "_\n",
- "B\n",
- "L\n",
- "O\n",
- "C\n",
- "K\n",
- " \n",
- " \n",
- "p\n",
- "y\n",
- "c\n",
- "o\n",
- "d"
+ "## Visualization"
]
},
{
"cell_type": "code",
- "execution_count": 10,
+ "execution_count": 12,
"metadata": {
"collapsed": false
},
@@ -2486,7 +2473,7 @@
},
{
"cell_type": "code",
- "execution_count": 11,
+ "execution_count": 13,
"metadata": {
"collapsed": false
},
@@ -2504,7 +2491,7 @@
},
{
"cell_type": "code",
- "execution_count": 12,
+ "execution_count": 14,
"metadata": {
"collapsed": false
},
@@ -2522,7 +2509,7 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": 15,
"metadata": {
"collapsed": false
},
@@ -2574,7 +2561,7 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": 16,
"metadata": {
"collapsed": false
},
@@ -2606,7 +2593,7 @@
},
{
"cell_type": "code",
- "execution_count": 15,
+ "execution_count": 17,
"metadata": {
"collapsed": false
},
@@ -2754,7 +2741,7 @@
},
{
"cell_type": "code",
- "execution_count": 16,
+ "execution_count": 18,
"metadata": {
"collapsed": false
},
@@ -2765,14 +2752,13 @@
"n_neurons_layer1 = 100\n",
"n_neurons_layer2 = 50\n",
"n_categories = 10\n",
- "\n",
"eta_vals = np.logspace(-5, 1, 7)\n",
"lmbd_vals = np.logspace(-5, 1, 7)"
]
},
{
"cell_type": "code",
- "execution_count": 17,
+ "execution_count": 19,
"metadata": {
"collapsed": false
},
@@ -2797,7 +2783,7 @@
},
{
"cell_type": "code",
- "execution_count": 18,
+ "execution_count": 20,
"metadata": {
"collapsed": false
},
@@ -2838,7 +2824,7 @@
},
{
"cell_type": "code",
- "execution_count": 19,
+ "execution_count": 21,
"metadata": {
"collapsed": false
},
@@ -2864,7 +2850,7 @@
},
{
"cell_type": "code",
- "execution_count": 20,
+ "execution_count": 22,
"metadata": {
"collapsed": false
},
@@ -2882,7 +2868,7 @@
},
{
"cell_type": "code",
- "execution_count": 21,
+ "execution_count": 23,
"metadata": {
"collapsed": false
},
@@ -2900,7 +2886,7 @@
},
{
"cell_type": "code",
- "execution_count": 22,
+ "execution_count": 24,
"metadata": {
"collapsed": false
},
@@ -2925,7 +2911,7 @@
},
{
"cell_type": "code",
- "execution_count": 23,
+ "execution_count": 25,
"metadata": {
"collapsed": false
},
@@ -2950,7 +2936,7 @@
},
{
"cell_type": "code",
- "execution_count": 24,
+ "execution_count": 26,
"metadata": {
"collapsed": false
},
diff --git a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz
index df4a2068c..e5b6f060d 100644
Binary files a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz and b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz differ
diff --git a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf
index 1f58cbc2a..2e8f83356 100644
Binary files a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf and b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf differ
diff --git a/doc/src/NeuralNet/NeuralNet.do.txt b/doc/src/NeuralNet/NeuralNet.do.txt
index 03bb895d7..23cb51eea 100644
--- a/doc/src/NeuralNet/NeuralNet.do.txt
+++ b/doc/src/NeuralNet/NeuralNet.do.txt
@@ -192,7 +192,7 @@ The output $y$ is produced via the activation function $f$
\]
!et
This function receives $x_i$ as inputs.
-Here the activation $z=\sum_{i=1}^n w_ix_i$.
+Here the activation $z=(\sum_{i=1}^n w_ix_i+b_i)$.
In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
@@ -312,7 +312,7 @@ is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively
We have that $\mathrm{W}_l$ is an $N_{l-1} \times N_l$ matrix, while $\hat{b}_l$ and $\hat{y}_l$ are $N_l \times 1$ column vectors.
With this notation, the sum becomes a matrix-vector multiplication, and we can write
-the equation for the activations of hidden layer 2 as
+the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as
!bt
\begin{equation}
\hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) =
@@ -489,7 +489,7 @@ o The input nodes pass values to the first hidden layer, its nodes pass the info
As a convention it is normal to call a network with one layer of input units, one layer of hidden
units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.
-For an MLP there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
+For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
Hereafter we will call the various entities of a layer for nodes.
There are also no connections within a single layer.
@@ -591,11 +591,10 @@ compact form as the matrix-vector products we discussed earlier,
\]
!et
-With the activation function $\hat{z}^l$ we can in turn define the
+With the activation values $\hat{z}^l$ we can in turn define the
output of layer $l$ as $\hat{a}^l = f(\hat{z}^l)$ where $f$ is our
activation function. In the examples here we will use the sigmoid
-function discussed in our logistic regression lectures and here as
-well. We will also use the same activation function $f$ for all layers
+function discussed in our logistic regression lectures. We will also use the same activation function $f$ for all layers
and their nodes. It means we have
!bt
@@ -680,7 +679,7 @@ and using the Hadamard product of two vectors we can write this as
!et
This is an important expression. The second term on the right handside
-measures how fast the cost is changing as a function of the $j$th
+measures how fast the cost function is changing as a function of the $j$th
output activation. If, for example, the cost function doesn't depend
much on a particular output node $j$, then $\delta_j^L$ will be small,
which is what we would expect. The first term on the right, measures
@@ -691,9 +690,9 @@ Notice that everything in the above equations is easily computed. In
particular, we compute $z_j^L$ while computing the behaviour of the
network, and it is only a small additional overhead to compute
$f'(z^L_j)$. The exact form of the derivative with respect to the
-outpuwill, of course, depend on the form of the cost function.
+output depends on the form of the cost function.
However, provided the cost function is known there should be little
-trouble computing
+trouble in calculating
!bt
\[
@@ -729,6 +728,7 @@ That is, the error $\delta_j^L$ is exactly equal to the rate of change of the co
===== Bringing it together =====
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
+
!bblock The starting equations
!bt
@@ -760,7 +760,7 @@ that it changes slowly when we minimize the weights via say gradient
descent. In this case we say the system learns slowly.
Another interesting feature is that is when the activation function,
-represented by the sigmoid function here, is rather flat when towards
+represented by the sigmoid function here, is rather flat when we move towards
its end values $0$ and $1$ (see the above Python codes). In these
cases, the derivatives of the activation function will also be close
to zero, meaning again that the gradients will be small and the
@@ -859,7 +859,7 @@ b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^L},
!eblock
The parameter $\eta$ is the learning parameter discussed in connection with the gradient descent methods.
-Here it is convenient to use stochastic radient descent with mini-batches with an outer loop that steps through multiple epochs of training.
+Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.
!split
@@ -880,11 +880,11 @@ also has other nice properties, such as a derivative that is simple to
calculate.
For an input $\boldsymbol{a}$ from the hidden layer, the probability that the input $\boldsymbol{x}$
-is in class 0 or 1 is just:
-
+is in class 0 or 1 is just. We let $\theta$ represent the unknown weights and biases to be adjusted by our equations). The variable $x$
+represents our activation values $z$. We have
!bt
\[
-P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{a}^T \boldsymbol{w}_{out})} ,
+P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{x}} ,
\]
!et
and
@@ -1165,7 +1165,7 @@ i.e. each neuron $j$ outputs the probability of being in class $j$ given an inpu
The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1.
The exponent is just the weighted sum of inputs as before:
-$$ z_j = \sum_{i=1}^n w_ {ij} a_i = \boldsymbol{a}^T \boldsymbol{w}_j .$$
+$$ z_j = \sum_{i=1}^n w_ {ij} a_i+b_j.$$
Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500
weights to the output layer.
@@ -1179,7 +1179,7 @@ or normal distribution. Setting all weights to zero means all neurons give the s
Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range
of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron $j$, $b_j$:
-$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j = \boldsymbol{a}^T \boldsymbol{w}_j + b_j .$$
+$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j.$$
The bias weights $\boldsymbol{b}$ are often initialized to zero, but a small value like $0.01$ ensures all neurons have some output which can be backpropagated in the first training cycle.
!bc pycod
@@ -1204,22 +1204,22 @@ output_bias = np.zeros(n_categories) + 0.01
===== Feed-forward pass =====
Denote $F$ the number of features, $H$ the number of hidden neurons and $C$ the number of categories.
-For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer:
+For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer $l$:
-$$ z_{j}^{h} = \sum_{i=1}^{F} w_{ij}^{h} x_i + b_{j}^{h} = \boldsymbol{x}^T \boldsymbol{w}_{j}^{h} + b_{j}^{h} ,$$
+$$ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$
this is then passed through our activation function
-$$ a_{j}^{h} = f(z_{j}^{h}) .$$
+$$ a_{j}^{l} = f(z_{j}^{l}) .$$
We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron $j$ in the output layer:
-$$ z_{j}^{o} = \sum_{i=1}^{H} w_{ij}^{o} a_{i}^{h} + b_{j}^{o} = (\boldsymbol{a}^{h})^T \boldsymbol{w}_{j}^{o} + b_{j}^{o} .$$
+$$ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$
Finally we calculate the output of neuron $j$ in the output layer using the softmax function:
-$$ a_{j}^{o} = \frac{\exp{(z_j^{o})}}
-{\sum_{c=0}^{C-1} \exp{(z_c^{o})}} .$$
+$$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
+{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .$$
!split
===== Matrix multiplication =====
@@ -1235,24 +1235,24 @@ and obtain a matrix that holds the weighted sum of inputs to the hidden layer
for each input image and each hidden neuron.
We also add the bias to obtain a matrix of weighted sums to the hidden layer $Z^{h}$:
-$$ Z^{h} = X W^{h} + B^{h} ,$$
+$$ \hat{z}^{l} = \hat{X} \hat{W}^{l} + \hat{b}^{l} ,$$
meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image.
This is then passed through the activation:
-$$ A^{h} = f(Z^h) .$$
+$$ \hat{a}^{l} = f(\hat{z}^l) .$$
This is fed to the output layer:
-$$ Z^{o} = A^{h} W^{o} + B^{o} .$$
+$$ \hat{z}^{L} = \hat{a}^{L} \hat{W}^{L} + \hat{b}^{L} .$$
Finally we receive our output values for each image and each category by passing it through the softmax function:
-$$ output = softmax (Z^{o}) = (n_{inputs}, n_{categories}) .$$
+$$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
!bc pycod
-# setup the feed-forward pass
+# setup the feed-forward pass, subscript h = hidden layer
def sigmoid(x):
return 1/(1 + np.exp(-x))
@@ -1386,32 +1386,32 @@ calculate the gradient efficently.
To more efficently train our network these equations are implemented using matrix operations.
The error in the output layer is calculated simply as
-$$ \Delta_o = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
+$$ \delta_L = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
The gradient for the output weights is calculated as
-$$ \nabla W_{o} = A^T \Delta_o = (n_{hidden}, n_{categories}) ,$$
+$$ \nabla W_{L} = \hat{a}^T \delta_L = (n_{hidden}, n_{categories}) ,$$
-where $A = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input.
+where $\hat{a} = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input.
Since we are going backwards we have to transpose the activation matrix.
The gradient with respect to the output bias is then
-$$ \nabla B_{o} = \sum_{i=1}^{n_{inputs}} \Delta_o = (n_{categories}) .$$
+$$ \nabla \hat{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .$$
The error in the hidden layer is
-$$ \Delta_h = \Delta_o W_{o}^T \circ f'(Z_{h}) = \Delta_o W_{o}^T \circ A_{h} \circ (1 - A_{h}) = (n_{inputs}, n_{hidden}) ,$$
+$$ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$
-where $f'(A_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean
+where $f'(a_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean
that we are summing up the products for each neuron in the output layer. The symbol $\circ$ denotes
the *Hadamard product*, meaning element-wise multiplication.
This again gives us the gradients in the hidden layer:
-$$ \nabla W_{h} = X^T \Delta_h = (n_{features}, n_{hidden}) ,$$
+$$ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,$$
-$$ \nabla B_{h} = \sum_{i=1}^{n_{inputs}} \Delta_h = (n_{hidden}) .$$
+$$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
!bc pycod
@@ -1644,17 +1644,14 @@ def accuracy_score_numpy(Y_test, Y_pred):
!ec
!split
-===== Adjust hyperparameters (if necessary, network architecture =====
+===== Adjust hyperparameters =====
We now perform a grid search to find the optimal hyperparameters for the network.
-Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98 \%$ ($2 \%$ error rate).
-
+Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\%$ ($2\%$ error rate).
!bc pycod
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
-!ec
-!bc pycod
# store the models for later use
DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
@@ -1674,10 +1671,13 @@ for i, eta in enumerate(eta_vals):
print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
print()
!ec
+
+!split
+===== Visualization =====
+
!bc pycod
-# optional
# visual representation of grid search
-# uses seaborn heatmap, I believe you can also do this with matplotlib imshow
+# uses seaborn heatmap, you can also do this with matplotlib imshow
import seaborn as sns
sns.set()
@@ -1714,14 +1714,20 @@ plt.show()
!split
===== scikit-learn implementation =====
-scikit-learn is a machine learning library for Python. It focuses more on traditional machine learning methods, such as regression, clustering, decision trees, etc. As such, it has only two types of neural networks: Multi Layer Perceptron outputting continuous values, *MPLRegressor*, and Multi Layer Perceptron outputting labels, *MLPClassifier*. We will see how simple it is to use these classes.
+_scikit-learn_ focuses more
+on traditional machine learning methods, such as regression,
+clustering, decision trees, etc. As such, it has only two types of
+neural networks: Multi Layer Perceptron outputting continuous values,
+*MPLRegressor*, and Multi Layer Perceptron outputting labels,
+*MLPClassifier*. We will see how simple it is to use these classes.
-scikit-learn implements a few improvements from our neural network, such as early stopping, a varying learning rate, different optimization methods, etc. We would therefore expect a better performance overall.
-
+_scikit-learn_ implements a few improvements from our neural network,
+such as early stopping, a varying learning rate, different
+optimization methods, etc. We would therefore expect a better
+performance overall.
!bc pycod
from sklearn.neural_network import MLPClassifier
-
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
@@ -1738,6 +1744,10 @@ for i, eta in enumerate(eta_vals):
print("Accuracy score on test set: ", dnn.score(X_test, Y_test))
print()
!ec
+
+
+!split
+===== Visualization =====
!bc pycod
# optional
# visual representation of grid search
@@ -2033,7 +2043,6 @@ batch_size = 100
n_neurons_layer1 = 100
n_neurons_layer2 = 50
n_categories = 10
-
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
!ec
@@ -2055,7 +2064,6 @@ for i, eta in enumerate(eta_vals):
print("Lambda = ", lmbd)
print("Test accuracy: %.3f" % DNN.test_accuracy)
print()
-
!ec
!bc pycod
# setup the feed-forward pass
+
# setup the feed-forward pass, subscript h = hidden layer
def sigmoid(x):
return 1/(1 + np.exp(-x))
@@ -329,7 +330,7 @@ predictions = predict(X_train)
Adjust hyperparameters (if necessary, network architecture
+Adjust hyperparameters
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
-
# store the models for later use
+# store the models for later use
DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
# grid search
@@ -266,44 +263,6 @@ DNN_numpy = np.
print()
# optional
-# visual representation of grid search
-# uses seaborn heatmap, I believe you can also do this with matplotlib imshow
-import seaborn as sns
-
-sns.set()
-
-train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
-test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
-
-for i in range(len(eta_vals)):
- for j in range(len(lmbd_vals)):
- dnn = DNN_numpy[i][j]
-
- train_pred = dnn.predict(X_train)
- test_pred = dnn.predict(X_test)
-
- train_accuracy[i][j] = accuracy_score(Y_train, train_pred)
- test_accuracy[i][j] = accuracy_score(Y_test, test_pred)
-
-
-fig, ax = plt.subplots(figsize = (10, 10))
-sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
-ax.set_title("Training Accuracy")
-ax.set_ylabel("$\eta$")
-ax.set_xlabel("$\lambda$")
-plt.show()
-
-fig, ax = plt.subplots(figsize = (10, 10))
-sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
-ax.set_title("Test Accuracy")
-ax.set_ylabel("$\eta$")
-ax.set_xlabel("$\lambda$")
-plt.show()
-
scikit-learn implementation
-
-Visualization
from sklearn.neural_network import MLPClassifier
-
-# store models for later use
-DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
-
-for i, eta in enumerate(eta_vals):
- for j, lmbd in enumerate(lmbd_vals):
- dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
- alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
- dnn.fit(X_train, Y_train)
-
- DNN_scikit[i][j] = dnn
-
- print("Learning rate = ", eta)
- print("Lambda = ", lmbd)
- print("Accuracy score on test set: ", dnn.score(X_test, Y_test))
- print()
-
# optional
-# visual representation of grid search
-# uses seaborn heatmap, could probably do this in matplotlib
+
# visual representation of grid search
+# uses seaborn heatmap, you can also do this with matplotlib imshow
import seaborn as sns
sns.set()
@@ -275,7 +248,7 @@ test_accuracy = npfor i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
- dnn = DNN_scikit[i][j]
+ dnn = DNN_numpy[i][j]
train_pred = dnn.predict(X_train)
test_pred = dnn.predict(X_test)
@@ -320,6 +293,8 @@ plt.show()
Building neural networks in Tensorflow and Keras
+scikit-learn implementation
from sklearn.neural_network import MLPClassifier
+# store models for later use
+DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+
+for i, eta in enumerate(eta_vals):
+ for j, lmbd in enumerate(lmbd_vals):
+ dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
+ alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
+ dnn.fit(X_train, Y_train)
+
+ DNN_scikit[i][j] = dnn
+
+ print("Learning rate = ", eta)
+ print("Lambda = ", lmbd)
+ print("Accuracy score on test set: ", dnn.score(X_test, Y_test))
+ print()
+
Tensorflow
-
-Visualization
pip3 install tensorflow
-
# optional
+# visual representation of grid search
+# uses seaborn heatmap, could probably do this in matplotlib
+import seaborn as sns
-
-
conda install tensorflow
+sns.set()
+
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+
+for i in range(len(eta_vals)):
+ for j in range(len(lmbd_vals)):
+ dnn = DNN_scikit[i][j]
+
+ train_pred = dnn.predict(X_train)
+ test_pred = dnn.predict(X_test)
+
+ train_accuracy[i][j] = accuracy_score(Y_train, train_pred)
+ test_accuracy[i][j] = accuracy_score(Y_test, test_pred)
+
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Test Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
Collect and pre-process data
+Building neural networks in Tensorflow and Keras
# import necessary packages
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn import datasets
-
-
-# ensure the same random numbers appear every time
-np.random.seed(0)
-
-# display images in notebook
-%matplotlib inline
-plt.rcParams['figure.figsize'] = (12,12)
-
-
-# download MNIST dataset
-digits = datasets.load_digits()
-
-# define inputs and labels
-inputs = digits.images
-labels = digits.target
-
-print("inputs = (n_inputs, pixel_width, pixel_height) = " + str(inputs.shape))
-print("labels = (n_inputs) = " + str(labels.shape))
-
-
-# flatten the image
-# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64
-n_inputs = len(inputs)
-inputs = inputs.reshape(n_inputs, -1)
-print("X = (n_inputs, n_features) = " + str(inputs.shape))
-
-
-# choose some random images to display
-indices = np.arange(n_inputs)
-random_indices = np.random.choice(indices, size=5)
-
-for i, image in enumerate(digits.images[random_indices]):
- plt.subplot(1, 5, i+1)
- plt.axis('off')
- plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
- plt.title("Label: %d" % digits.target[random_indices[i]])
-plt.show()
-
from keras.utils import to_categorical
-from sklearn.model_selection import train_test_split
-
-# one-hot representation of labels
-labels = to_categorical(labels)
-
-# split into train and test data
-train_size = 0.8
-test_size = 1 - train_size
-X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,
- test_size=test_size)
-
Using TensorFlow backend
+Tensorflow
-
-
+import tensorflow as tf
+
pip3 install tensorflow
+
conda install tensorflow
Optimizing and using gradient descent
+Collect and pre-process data
epochs = 100
-batch_size = 100
-n_neurons_layer1 = 100
-n_neurons_layer2 = 50
-n_categories = 10
+
# import necessary packages
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn import datasets
-eta_vals = np.logspace(-5, 1, 7)
-lmbd_vals = np.logspace(-5, 1, 7)
-
DNN_tf = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
-
-for i, eta in enumerate(eta_vals):
- for j, lmbd in enumerate(lmbd_vals):
- DNN = NeuralNetworkTensorflow(X_train, Y_train, X_test, Y_test,
- n_neurons_layer1, n_neurons_layer2, n_categories,
- epochs=epochs, batch_size=batch_size, eta=eta, lmbd=lmbd)
- DNN.fit()
-
- DNN_tf[i][j] = DNN
-
- print("Learning rate = ", eta)
- print("Lambda = ", lmbd)
- print("Test accuracy: %.3f" % DNN.test_accuracy)
- print()
-
-
# optional
-# visual representation of grid search
-# uses seaborn heatmap, could probably do this in matplotlib
-import seaborn as sns
+# display images in notebook
+%matplotlib inline
+plt.rcParams['figure.figsize'] = (12,12)
-sns.set()
-train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
-test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+# download MNIST dataset
+digits = datasets.load_digits()
-for i in range(len(eta_vals)):
- for j in range(len(lmbd_vals)):
- DNN = DNN_tf[i][j]
+# define inputs and labels
+inputs = digits.images
+labels = digits.target
- train_accuracy[i][j] = DNN.train_accuracy
- test_accuracy[i][j] = DNN.test_accuracy
+print("inputs = (n_inputs, pixel_width, pixel_height) = " + str(inputs.shape))
+print("labels = (n_inputs) = " + str(labels.shape))
-
-fig, ax = plt.subplots(figsize = (10, 10))
-sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
-ax.set_title("Training Accuracy")
-ax.set_ylabel("$\eta$")
-ax.set_xlabel("$\lambda$")
-plt.show()
-fig, ax = plt.subplots(figsize = (10, 10))
-sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
-ax.set_title("Test Accuracy")
-ax.set_ylabel("$\eta$")
-ax.set_xlabel("$\lambda$")
+# flatten the image
+# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64
+n_inputs = len(inputs)
+inputs = inputs.reshape(n_inputs, -1)
+print("X = (n_inputs, n_features) = " + str(inputs.shape))
+
+
+# choose some random images to display
+indices = np.arange(n_inputs)
+random_indices = np.random.choice(indices, size=5)
+
+for i, image in enumerate(digits.images[random_indices]):
+ plt.subplot(1, 5, i+1)
+ plt.axis('off')
+ plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
+ plt.title("Label: %d" % digits.target[random_indices[i]])
plt.show()
# optional
-# we can use log files to visualize our graph in Tensorboard
-writer = tf.summary.FileWriter('logs/')
-writer.add_graph(tf.get_default_graph())
+
from keras.utils import to_categorical
+from sklearn.model_selection import train_test_split
+
+# one-hot representation of labels
+labels = to_categorical(labels)
+
+# split into train and test data
+train_size = 0.8
+test_size = 1 - train_size
+X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,
+ test_size=test_size)
Using Keras
+Using TensorFlow backend
-conda install keras
-
+
pip3 install keras
-
import tensorflow as tf
-
diff --git a/doc/pub/NeuralNet/html/NeuralNet-bs.html b/doc/pub/NeuralNet/html/NeuralNet-bs.html
index da168971c..ed51ffa10 100644
--- a/doc/pub/NeuralNet/html/NeuralNet-bs.html
+++ b/doc/pub/NeuralNet/html/NeuralNet-bs.html
@@ -110,20 +110,19 @@ Automatically generated HTML file from DocOnce source
('Improving performance', 2, None, '___sec44'),
('Full object-oriented implementation', 2, None, '___sec45'),
('Evaluate model performance on test data', 2, None, '___sec46'),
- ('Adjust hyperparameters (if necessary, network architecture',
- 2,
- None,
- '___sec47'),
- ('scikit-learn implementation', 2, None, '___sec48'),
+ ('Adjust hyperparameters', 2, None, '___sec47'),
+ ('Visualization', 2, None, '___sec48'),
+ ('scikit-learn implementation', 2, None, '___sec49'),
+ ('Visualization', 2, None, '___sec50'),
('Building neural networks in Tensorflow and Keras',
2,
None,
- '___sec49'),
- ('Tensorflow', 2, None, '___sec50'),
- ('Collect and pre-process data', 2, None, '___sec51'),
- ('Using TensorFlow backend', 2, None, '___sec52'),
- ('Optimizing and using gradient descent', 2, None, '___sec53'),
- ('Using Keras', 2, None, '___sec54')]}
+ '___sec51'),
+ ('Tensorflow', 2, None, '___sec52'),
+ ('Collect and pre-process data', 2, None, '___sec53'),
+ ('Using TensorFlow backend', 2, None, '___sec54'),
+ ('Optimizing and using gradient descent', 2, None, '___sec55'),
+ ('Using Keras', 2, None, '___sec56')]}
end of tocinfo -->
@@ -208,14 +207,16 @@ MathJax.Hub.Config({
from keras.models import Sequential
-from keras.layers import Dense
-from keras.regularizers import l2
-from keras.optimizers import SGD
-
-def create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories, eta, lmbd):
- model = Sequential()
- model.add(Dense(n_neurons_layer1, activation='sigmoid', kernel_regularizer=l2(lmbd)))
- model.add(Dense(n_neurons_layer2, activation='sigmoid', kernel_regularizer=l2(lmbd)))
- model.add(Dense(n_categories, activation='softmax'))
+class NeuralNetworkTensorflow:
+ def __init__(
+ self,
+ X_train,
+ Y_train,
+ X_test,
+ Y_test,
+ n_neurons_layer1=100,
+ n_neurons_layer2=50,
+ n_categories=2,
+ epochs=10,
+ batch_size=100,
+ eta=0.1,
+ lmbd=0.0,
+ ):
+
+ # keep track of number of steps
+ self.global_step = tf.Variable(0, dtype=tf.int32, trainable=False, name='global_step')
+
+ self.X_train = X_train
+ self.Y_train = Y_train
+ self.X_test = X_test
+ self.Y_test = Y_test
+
+ self.n_inputs = X_train.shape[0]
+ self.n_features = X_train.shape[1]
+ self.n_neurons_layer1 = n_neurons_layer1
+ self.n_neurons_layer2 = n_neurons_layer2
+ self.n_categories = n_categories
+
+ self.epochs = epochs
+ self.batch_size = batch_size
+ self.iterations = self.n_inputs // self.batch_size
+ self.eta = eta
+ self.lmbd = lmbd
+
+ # build network piece by piece
+ # name scopes (with) are used to enforce creation of new variables
+ # https://www.tensorflow.org/guide/variables
+ self.create_placeholders()
+ self.create_DNN()
+ self.create_loss()
+ self.create_optimiser()
+ self.create_accuracy()
- sgd = SGD(lr=eta)
- model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
+ def create_placeholders(self):
+ # placeholders are fine here, but "Datasets" are the preferred method
+ # of streaming data into a model
+ with tf.name_scope('data'):
+ self.X = tf.placeholder(tf.float32, shape=(None, self.n_features), name='X_data')
+ self.Y = tf.placeholder(tf.float32, shape=(None, self.n_categories), name='Y_data')
- return model
+ def create_DNN(self):
+ with tf.name_scope('DNN'):
+ # the weights are stored to calculate regularization loss later
+
+ # Fully connected layer 1
+ self.W_fc1 = self.weight_variable([self.n_features, self.n_neurons_layer1], name='fc1', dtype=tf.float32)
+ b_fc1 = self.bias_variable([self.n_neurons_layer1], name='fc1', dtype=tf.float32)
+ a_fc1 = tf.nn.sigmoid(tf.matmul(self.X, self.W_fc1) + b_fc1)
+
+ # Fully connected layer 2
+ self.W_fc2 = self.weight_variable([self.n_neurons_layer1, self.n_neurons_layer2], name='fc2', dtype=tf.float32)
+ b_fc2 = self.bias_variable([self.n_neurons_layer2], name='fc2', dtype=tf.float32)
+ a_fc2 = tf.nn.sigmoid(tf.matmul(a_fc1, self.W_fc2) + b_fc2)
+
+ # Output layer
+ self.W_out = self.weight_variable([self.n_neurons_layer2, self.n_categories], name='out', dtype=tf.float32)
+ b_out = self.bias_variable([self.n_categories], name='out', dtype=tf.float32)
+ self.z_out = tf.matmul(a_fc2, self.W_out) + b_out
+
+ def create_loss(self):
+ with tf.name_scope('loss'):
+ softmax_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=self.Y, logits=self.z_out))
+
+ regularizer_loss_fc1 = tf.nn.l2_loss(self.W_fc1)
+ regularizer_loss_fc2 = tf.nn.l2_loss(self.W_fc2)
+ regularizer_loss_out = tf.nn.l2_loss(self.W_out)
+ regularizer_loss = self.lmbd*(regularizer_loss_fc1 + regularizer_loss_fc2 + regularizer_loss_out)
+
+ self.loss = softmax_loss + regularizer_loss
+
+ def create_accuracy(self):
+ with tf.name_scope('accuracy'):
+ probabilities = tf.nn.softmax(self.z_out)
+ predictions = tf.argmax(probabilities, axis=1)
+ labels = tf.argmax(self.Y, axis=1)
+
+ correct_predictions = tf.equal(predictions, labels)
+ correct_predictions = tf.cast(correct_predictions, tf.float32)
+ self.accuracy = tf.reduce_mean(correct_predictions)
+
+ def create_optimiser(self):
+ with tf.name_scope('optimizer'):
+ self.optimizer = tf.train.GradientDescentOptimizer(learning_rate=self.eta).minimize(self.loss, global_step=self.global_step)
+
+ def weight_variable(self, shape, name='', dtype=tf.float32):
+ initial = tf.truncated_normal(shape, stddev=0.1)
+ return tf.Variable(initial, name=name, dtype=dtype)
+
+ def bias_variable(self, shape, name='', dtype=tf.float32):
+ initial = tf.constant(0.1, shape=shape)
+ return tf.Variable(initial, name=name, dtype=dtype)
+
+ def fit(self):
+ data_indices = np.arange(self.n_inputs)
+
+ with tf.Session() as sess:
+ sess.run(tf.global_variables_initializer())
+ for i in range(self.epochs):
+ for j in range(self.iterations):
+ chosen_datapoints = np.random.choice(data_indices, size=self.batch_size, replace=False)
+ batch_X, batch_Y = self.X_train[chosen_datapoints], self.Y_train[chosen_datapoints]
+
+ sess.run([DNN.loss, DNN.optimizer],
+ feed_dict={DNN.X: batch_X,
+ DNN.Y: batch_Y})
+ accuracy = sess.run(DNN.accuracy,
+ feed_dict={DNN.X: batch_X,
+ DNN.Y: batch_Y})
+ step = sess.run(DNN.global_step)
+
+ self.train_loss, self.train_accuracy = sess.run([DNN.loss, DNN.accuracy],
+ feed_dict={DNN.X: self.X_train,
+ DNN.Y: self.Y_train})
+
+ self.test_loss, self.test_accuracy = sess.run([DNN.loss, DNN.accuracy],
+ feed_dict={DNN.X: self.X_test,
+ DNN.Y: self.Y_test})
DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
-
-for i, eta in enumerate(eta_vals):
- for j, lmbd in enumerate(lmbd_vals):
- DNN = create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories,
- eta=eta, lmbd=lmbd)
- DNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)
- scores = DNN.evaluate(X_test, Y_test)
-
- DNN_keras[i][j] = DNN
-
- print("Learning rate = ", eta)
- print("Lambda = ", lmbd)
- print("Test accuracy: %.3f" % scores[1])
- print()
-
# optional
-# visual representation of grid search
-# uses seaborn heatmap, could probably do this in matplotlib
-import seaborn as sns
-
-sns.set()
-
-train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
-test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
-
-for i in range(len(eta_vals)):
- for j in range(len(lmbd_vals)):
- DNN = DNN_keras[i][j]
-
- train_accuracy[i][j] = DNN.evaluate(X_train, Y_train)[1]
- test_accuracy[i][j] = DNN.evaluate(X_test, Y_test)[1]
-
-
-fig, ax = plt.subplots(figsize = (10, 10))
-sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
-ax.set_title("Training Accuracy")
-ax.set_ylabel("$\eta$")
-ax.set_xlabel("$\lambda$")
-plt.show()
-
-fig, ax = plt.subplots(figsize = (10, 10))
-sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
-ax.set_title("Test Accuracy")
-ax.set_ylabel("$\eta$")
-ax.set_xlabel("$\lambda$")
-plt.show()
-
Oct 2, 2018
Oct 4, 2018
-Oct 2, 2018
Oct 4, 2018
This function receives \( x_i \) as inputs.
-Here the activation \( z=\sum_{i=1}^n w_ix_i \).
+Here the activation \( z=(\sum_{i=1}^n w_ix_i+b_i) \).
In an FFNN of such neurons, the inputs \( x_i \) are the outputs of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
@@ -549,7 +549,7 @@ is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l
$$
\begin{equation}
@@ -761,7 +761,7 @@ As a convention it is normal to call a network with one layer of input units,
units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.
@@ -979,7 +978,7 @@ $$
$$
@@ -1039,6 +1038,8 @@ That is, the error \( \delta_j^L \) is exactly equal to the rate of change of th
@@ -1082,7 +1083,7 @@ descent. In this case we say the system learns slowly.
$$
-P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{a}^T \boldsymbol{w}_{out})} ,
+P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{x}} ,
$$
@@ -1589,7 +1590,7 @@ The denominator is a normalization factor to ensure the outputs (probabilities)
The exponent is just the weighted sum of inputs as before:
-$$ z_j = \sum_{i=1}^n w_ {ij} a_i = \boldsymbol{a}^T \boldsymbol{w}_j .$$
+$$ z_j = \sum_{i=1}^n w_ {ij} a_i+b_j.$$
-$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j = \boldsymbol{a}^T \boldsymbol{w}_j + b_j .$$
+$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j.$$
-$$ z_{j}^{h} = \sum_{i=1}^{F} w_{ij}^{h} x_i + b_{j}^{h} = \boldsymbol{x}^T \boldsymbol{w}_{j}^{h} + b_{j}^{h} ,$$
+$$ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$
-$$ a_{j}^{h} = f(z_{j}^{h}) .$$
+$$ a_{j}^{l} = f(z_{j}^{l}) .$$
-$$ z_{j}^{o} = \sum_{i=1}^{H} w_{ij}^{o} a_{i}^{h} + b_{j}^{o} = (\boldsymbol{a}^{h})^T \boldsymbol{w}_{j}^{o} + b_{j}^{o} .$$
+$$ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$
-$$ a_{j}^{o} = \frac{\exp{(z_j^{o})}}
-{\sum_{c=0}^{C-1} \exp{(z_c^{o})}} .$$
+$$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
+{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .$$
@@ -1691,7 +1692,7 @@ for each input image and each hidden neuron.
We also add the bias to obtain a matrix of weighted sums to the hidden layer \( Z^{h} \):
-$$ Z^{h} = X W^{h} + B^{h} ,$$
+$$ \hat{z}^{l} = \hat{X} \hat{W}^{l} + \hat{b}^{l} ,$$
-$$ A^{h} = f(Z^h) .$$
+$$ \hat{a}^{l} = f(\hat{z}^l) .$$
-$$ Z^{o} = A^{h} W^{o} + B^{o} .$$
+$$ \hat{z}^{L} = \hat{a}^{L} \hat{W}^{L} + \hat{b}^{L} .$$
-$$ output = softmax (Z^{o}) = (n_{inputs}, n_{categories}) .$$
+$$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
# setup the feed-forward pass
+
# setup the feed-forward pass, subscript h = hidden layer
def sigmoid(x):
return 1/(1 + np.exp(-x))
@@ -1892,36 +1893,36 @@ To more efficently train our network these equations are implemented using matri
The error in the output layer is calculated simply as
-$$ \Delta_o = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
+$$ \delta_L = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
-$$ \nabla W_{o} = A^T \Delta_o = (n_{hidden}, n_{categories}) ,$$
+$$ \nabla W_{L} = \hat{a}^T \delta_L = (n_{hidden}, n_{categories}) ,$$
-$$ \nabla B_{o} = \sum_{i=1}^{n_{inputs}} \Delta_o = (n_{categories}) .$$
+$$ \nabla \hat{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .$$
-$$ \Delta_h = \Delta_o W_{o}^T \circ f'(Z_{h}) = \Delta_o W_{o}^T \circ A_{h} \circ (1 - A_{h}) = (n_{inputs}, n_{hidden}) ,$$
+$$ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$
-$$ \nabla W_{h} = X^T \Delta_h = (n_{features}, n_{hidden}) ,$$
+$$ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,$$
-$$ \nabla B_{h} = \sum_{i=1}^{n_{inputs}} \Delta_h = (n_{hidden}) .$$
+$$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
Adjust hyperparameters (if necessary, network architecture
+Adjust hyperparameters
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
-
# store the models for later use
+# store the models for later use
DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
# grid search
@@ -2221,12 +2218,17 @@ DNN_numpy = np.zeros((len(eta_vals), print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
print()
Visualization
+
# optional
-# visual representation of grid search
-# uses seaborn heatmap, I believe you can also do this with matplotlib imshow
+
# visual representation of grid search
+# uses seaborn heatmap, you can also do this with matplotlib imshow
import seaborn as sns
sns.set()
@@ -2263,19 +2265,26 @@ plt.show()
scikit-learn implementation
+scikit-learn implementation
from sklearn.neural_network import MLPClassifier
-
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
@@ -2292,6 +2301,11 @@ DNN_scikit = np.zeros((len(eta_vals), print("Accuracy score on test set: ", dnn.score(X_test, Y_test))
print()
Visualization
Building neural networks in Tensorflow and Keras
+Building neural networks in Tensorflow and Keras
Tensorflow
+Tensorflow
Collect and pre-process data
+Collect and pre-process data
Using TensorFlow backend
+Using TensorFlow backend
Optimizing and using gradient descent
+Optimizing and using gradient descent
Using Keras
+Using Keras
Oct 2, 2018
Oct 4, 2018
@@ -413,7 +412,7 @@ $$
$$
This function receives \( x_i \) as inputs.
-Here the activation \( z=\sum_{i=1}^n w_ix_i \).
+Here the activation \( z=(\sum_{i=1}^n w_ix_i+b_i) \).
In an FFNN of such neurons, the inputs \( x_i \) are the outputs of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
@@ -560,7 +559,7 @@ is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l
# setup the feed-forward pass
+
# setup the feed-forward pass, subscript h = hidden layer
def sigmoid(x):
return 1/(1 + np.exp(-x))
@@ -1759,38 +1759,38 @@ calculate the gradient efficently.
To more efficently train our network these equations are implemented using matrix operations.
The error in the output layer is calculated simply as
-$$ \Delta_o = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
+$$ \delta_L = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
-Adjust hyperparameters (if necessary, network architecture
+Adjust hyperparameters
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
-
# store the models for later use
+# store the models for later use
DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
# grid search
@@ -2072,12 +2068,16 @@ DNN_numpy = np.zeros((len(eta_vals), print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
print()
+
+Visualization
+
# optional
-# visual representation of grid search
-# uses seaborn heatmap, I believe you can also do this with matplotlib imshow
+
# visual representation of grid search
+# uses seaborn heatmap, you can also do this with matplotlib imshow
import seaborn as sns
sns.set()
@@ -2113,19 +2113,26 @@ plt.show()
-scikit-learn implementation
+scikit-learn implementation
from sklearn.neural_network import MLPClassifier
-
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
@@ -2143,6 +2150,10 @@ DNN_scikit = np.zeros((len(eta_vals), print()
+
+Visualization
+# optional
@@ -2183,7 +2194,7 @@ plt.show()
-Building neural networks in Tensorflow and Keras
+Building neural networks in Tensorflow and Keras
-Tensorflow
+Tensorflow
-Collect and pre-process data
+Collect and pre-process data
-Using TensorFlow backend
+Using TensorFlow backend
-Optimizing and using gradient descent
+Optimizing and using gradient descent
-Using Keras
+Using Keras
Oct 2, 2018
Oct 4, 2018
@@ -418,7 +417,7 @@ $$
$$
This function receives \( x_i \) as inputs.
-Here the activation \( z=\sum_{i=1}^n w_ix_i \).
+Here the activation \( z=(\sum_{i=1}^n w_ix_i+b_i) \).
In an FFNN of such neurons, the inputs \( x_i \) are the outputs of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
@@ -565,7 +564,7 @@ is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l
# setup the feed-forward pass
+
# setup the feed-forward pass, subscript h = hidden layer
def sigmoid(x):
return 1/(1 + np.exp(-x))
@@ -1764,38 +1764,38 @@ calculate the gradient efficently.
To more efficently train our network these equations are implemented using matrix operations.
The error in the output layer is calculated simply as
-$$ \Delta_o = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
+$$ \delta_L = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
Adjust hyperparameters
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
-
# store the models for later use
+# store the models for later use
DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
# grid search
@@ -2077,12 +2073,16 @@ DNN_numpy = np.
print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
print()
+
+Visualization
+
# optional
-# visual representation of grid search
-# uses seaborn heatmap, I believe you can also do this with matplotlib imshow
+
# visual representation of grid search
+# uses seaborn heatmap, you can also do this with matplotlib imshow
import seaborn as sns
sns.set()
@@ -2118,19 +2118,26 @@ plt.show()
-scikit-learn implementation
+scikit-learn implementation
from sklearn.neural_network import MLPClassifier
-
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
@@ -2148,6 +2155,10 @@ DNN_scikit = np
print()
+
+Visualization
+# optional
@@ -2188,7 +2199,7 @@ plt.show()
-Building neural networks in Tensorflow and Keras
+Building neural networks in Tensorflow and Keras
-Tensorflow
+Tensorflow
-Collect and pre-process data
+Collect and pre-process data
-Using TensorFlow backend
+Using TensorFlow backend
-Optimizing and using gradient descent
+Optimizing and using gradient descent
Using Keras