lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated typos

Browse Source
This commit is contained in:
mhjensen
2018-10-04 05:35:51 +02:00
parent ff08e04b6e
commit 0c44eedada
64 changed files with 1948 additions and 2013 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/NeuralNet/NeuralNet.do.txt
+56 -48
View File
@@ -192,7 +192,7 @@ The output $y$ is produced via the activation function $f$
\]
!et
This function receives $x_i$ as inputs.
Here the activation $z=\sum_{i=1}^n w_ix_i$.
Here the activation $z=(\sum_{i=1}^n w_ix_i+b_i)$.
In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
@@ -312,7 +312,7 @@ is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively
We have that $\mathrm{W}_l$ is an $N_{l-1} \times N_l$ matrix, while $\hat{b}_l$ and $\hat{y}_l$ are $N_l \times 1$ column vectors.
With this notation, the sum becomes a matrix-vector multiplication, and we can write
the equation for the activations of hidden layer 2 as
the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as
!bt
\begin{equation}
\hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) =
@@ -489,7 +489,7 @@ o The input nodes pass values to the first hidden layer, its nodes pass the info
As a convention it is normal to call a network with one layer of input units, one layer of hidden
units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.
For an MLP there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
Hereafter we will call the various entities of a layer for nodes.
There are also no connections within a single layer.
@@ -591,11 +591,10 @@ compact form as the matrix-vector products we discussed earlier,
\]
!et
With the activation function $\hat{z}^l$ we can in turn define the
With the activation values $\hat{z}^l$ we can in turn define the
output of layer $l$ as $\hat{a}^l = f(\hat{z}^l)$ where $f$ is our
activation function. In the examples here we will use the sigmoid
function discussed in our logistic regression lectures and here as
well. We will also use the same activation function $f$ for all layers
function discussed in our logistic regression lectures. We will also use the same activation function $f$ for all layers
and their nodes. It means we have
!bt
@@ -680,7 +679,7 @@ and using the Hadamard product of two vectors we can write this as
!et
This is an important expression. The second term on the right handside
measures how fast the cost is changing as a function of the $j$th
measures how fast the cost function is changing as a function of the $j$th
output activation. If, for example, the cost function doesn't depend
much on a particular output node $j$, then $\delta_j^L$ will be small,
which is what we would expect. The first term on the right, measures
@@ -691,9 +690,9 @@ Notice that everything in the above equations is easily computed. In
particular, we compute $z_j^L$ while computing the behaviour of the
network, and it is only a small additional overhead to compute
$f'(z^L_j)$. The exact form of the derivative with respect to the
outpuwill, of course, depend on the form of the cost function.
output depends on the form of the cost function.
However, provided the cost function is known there should be little
trouble computing
trouble in calculating
!bt
\[
@@ -729,6 +728,7 @@ That is, the error $\delta_j^L$ is exactly equal to the rate of change of the co
===== Bringing it together =====
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
!bblock The starting equations
!bt
@@ -760,7 +760,7 @@ that it changes slowly when we minimize the weights via say gradient
descent. In this case we say the system learns slowly.
Another interesting feature is that is when the activation function,
represented by the sigmoid function here, is rather flat when towards
represented by the sigmoid function here, is rather flat when we move towards
its end values $0$ and $1$ (see the above Python codes). In these
cases, the derivatives of the activation function will also be close
to zero, meaning again that the gradients will be small and the
@@ -859,7 +859,7 @@ b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^L},
!eblock
The parameter $\eta$ is the learning parameter discussed in connection with the gradient descent methods.
Here it is convenient to use stochastic radient descent with mini-batches with an outer loop that steps through multiple epochs of training.
Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.
!split
@@ -880,11 +880,11 @@ also has other nice properties, such as a derivative that is simple to
calculate.
For an input $\boldsymbol{a}$ from the hidden layer, the probability that the input $\boldsymbol{x}$
is in class 0 or 1 is just:
is in class 0 or 1 is just. We let $\theta$ represent the unknown weights and biases to be adjusted by our equations). The variable $x$
represents our activation values $z$. We have
!bt
\[
P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{a}^T \boldsymbol{w}_{out})} ,
P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{x}} ,
\]
!et
and
@@ -1165,7 +1165,7 @@ i.e. each neuron $j$ outputs the probability of being in class $j$ given an inpu
The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1.
The exponent is just the weighted sum of inputs as before:
$$ z_j = \sum_{i=1}^n w_ {ij} a_i = \boldsymbol{a}^T \boldsymbol{w}_j .$$
$$ z_j = \sum_{i=1}^n w_ {ij} a_i+b_j.$$
Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500
weights to the output layer.
@@ -1179,7 +1179,7 @@ or normal distribution. Setting all weights to zero means all neurons give the s
Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range
of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron $j$, $b_j$:
$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j = \boldsymbol{a}^T \boldsymbol{w}_j + b_j .$$
$$ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j.$$
The bias weights $\boldsymbol{b}$ are often initialized to zero, but a small value like $0.01$ ensures all neurons have some output which can be backpropagated in the first training cycle.
!bc pycod
@@ -1204,22 +1204,22 @@ output_bias = np.zeros(n_categories) + 0.01
===== Feed-forward pass =====
Denote $F$ the number of features, $H$ the number of hidden neurons and $C$ the number of categories.
For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer:
For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer $l$:
$$ z_{j}^{h} = \sum_{i=1}^{F} w_{ij}^{h} x_i + b_{j}^{h} = \boldsymbol{x}^T \boldsymbol{w}_{j}^{h} + b_{j}^{h} ,$$
$$ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$
this is then passed through our activation function
$$ a_{j}^{h} = f(z_{j}^{h}) .$$
$$ a_{j}^{l} = f(z_{j}^{l}) .$$
We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron $j$ in the output layer:
$$ z_{j}^{o} = \sum_{i=1}^{H} w_{ij}^{o} a_{i}^{h} + b_{j}^{o} = (\boldsymbol{a}^{h})^T \boldsymbol{w}_{j}^{o} + b_{j}^{o} .$$
$$ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$
Finally we calculate the output of neuron $j$ in the output layer using the softmax function:
$$ a_{j}^{o} = \frac{\exp{(z_j^{o})}}
{\sum_{c=0}^{C-1} \exp{(z_c^{o})}} .$$
$$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .$$
!split
===== Matrix multiplication =====
@@ -1235,24 +1235,24 @@ and obtain a matrix that holds the weighted sum of inputs to the hidden layer
for each input image and each hidden neuron.
We also add the bias to obtain a matrix of weighted sums to the hidden layer $Z^{h}$:
$$ Z^{h} = X W^{h} + B^{h} ,$$
$$ \hat{z}^{l} = \hat{X} \hat{W}^{l} + \hat{b}^{l} ,$$
meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image.
This is then passed through the activation:
$$ A^{h} = f(Z^h) .$$
$$ \hat{a}^{l} = f(\hat{z}^l) .$$
This is fed to the output layer:
$$ Z^{o} = A^{h} W^{o} + B^{o} .$$
$$ \hat{z}^{L} = \hat{a}^{L} \hat{W}^{L} + \hat{b}^{L} .$$
Finally we receive our output values for each image and each category by passing it through the softmax function:
$$ output = softmax (Z^{o}) = (n_{inputs}, n_{categories}) .$$
$$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
!bc pycod
# setup the feed-forward pass
# setup the feed-forward pass, subscript h = hidden layer
def sigmoid(x):
return 1/(1 + np.exp(-x))
@@ -1386,32 +1386,32 @@ calculate the gradient efficently.
To more efficently train our network these equations are implemented using matrix operations.
The error in the output layer is calculated simply as
$$ \Delta_o = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
$$ \delta_L = \hat{y} - y = (n_{inputs}, n_{categories}) .$$
The gradient for the output weights is calculated as
$$ \nabla W_{o} = A^T \Delta_o = (n_{hidden}, n_{categories}) ,$$
$$ \nabla W_{L} = \hat{a}^T \delta_L = (n_{hidden}, n_{categories}) ,$$
where $A = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input.
where $\hat{a} = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input.
Since we are going backwards we have to transpose the activation matrix.
The gradient with respect to the output bias is then
$$ \nabla B_{o} = \sum_{i=1}^{n_{inputs}} \Delta_o = (n_{categories}) .$$
$$ \nabla \hat{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .$$
The error in the hidden layer is
$$ \Delta_h = \Delta_o W_{o}^T \circ f'(Z_{h}) = \Delta_o W_{o}^T \circ A_{h} \circ (1 - A_{h}) = (n_{inputs}, n_{hidden}) ,$$
$$ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$
where $f'(A_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean
where $f'(a_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean
that we are summing up the products for each neuron in the output layer. The symbol $\circ$ denotes
the *Hadamard product*, meaning element-wise multiplication.
This again gives us the gradients in the hidden layer:
$$ \nabla W_{h} = X^T \Delta_h = (n_{features}, n_{hidden}) ,$$
$$ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,$$
$$ \nabla B_{h} = \sum_{i=1}^{n_{inputs}} \Delta_h = (n_{hidden}) .$$
$$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
!bc pycod
@@ -1644,17 +1644,14 @@ def accuracy_score_numpy(Y_test, Y_pred):
!ec
!split
===== Adjust hyperparameters (if necessary, network architecture =====
===== Adjust hyperparameters =====
We now perform a grid search to find the optimal hyperparameters for the network.
Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98 \%$ ($2 \%$ error rate).
Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\%$ ($2\%$ error rate).
!bc pycod
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
!ec
!bc pycod
# store the models for later use
DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
@@ -1674,10 +1671,13 @@ for i, eta in enumerate(eta_vals):
print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
print()
!ec
!split
===== Visualization =====
!bc pycod
# optional
# visual representation of grid search
# uses seaborn heatmap, I believe you can also do this with matplotlib imshow
# uses seaborn heatmap, you can also do this with matplotlib imshow
import seaborn as sns
sns.set()
@@ -1714,14 +1714,20 @@ plt.show()
!split
===== scikit-learn implementation =====
scikit-learn is a machine learning library for Python. It focuses more on traditional machine learning methods, such as regression, clustering, decision trees, etc. As such, it has only two types of neural networks: Multi Layer Perceptron outputting continuous values, *MPLRegressor*, and Multi Layer Perceptron outputting labels, *MLPClassifier*. We will see how simple it is to use these classes.
_scikit-learn_ focuses more
on traditional machine learning methods, such as regression,
clustering, decision trees, etc. As such, it has only two types of
neural networks: Multi Layer Perceptron outputting continuous values,
*MPLRegressor*, and Multi Layer Perceptron outputting labels,
*MLPClassifier*. We will see how simple it is to use these classes.
scikit-learn implements a few improvements from our neural network, such as early stopping, a varying learning rate, different optimization methods, etc. We would therefore expect a better performance overall.
_scikit-learn_ implements a few improvements from our neural network,
such as early stopping, a varying learning rate, different
optimization methods, etc. We would therefore expect a better
performance overall.
!bc pycod
from sklearn.neural_network import MLPClassifier
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
@@ -1738,6 +1744,10 @@ for i, eta in enumerate(eta_vals):
print("Accuracy score on test set: ", dnn.score(X_test, Y_test))
print()
!ec
!split
===== Visualization =====
!bc pycod
# optional
# visual representation of grid search
@@ -2033,7 +2043,6 @@ batch_size = 100
n_neurons_layer1 = 100
n_neurons_layer2 = 50
n_categories = 10
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
!ec
@@ -2055,7 +2064,6 @@ for i, eta in enumerate(eta_vals):
print("Lambda = ", lmbd)
print("Test accuracy: %.3f" % DNN.test_accuracy)
print()
!ec
!bc pycod