small update on neural networks
This commit is contained in:
mhjensen
@@ -92,54 +92,55 @@ Automatically generated HTML file from DocOnce source
|
||||
'___sec30'),
|
||||
('Defining the cost function', 2, None, '___sec31'),
|
||||
('Example: binary classification problem', 2, None, '___sec32'),
|
||||
('The Softmax function', 2, None, '___sec33'),
|
||||
('Developing a code for doing neural networks with back '
|
||||
'propagation',
|
||||
2,
|
||||
None,
|
||||
'___sec33'),
|
||||
('Collect and pre-process data', 2, None, '___sec34'),
|
||||
('Train and test datasets', 2, None, '___sec35'),
|
||||
('Define model and architecture', 2, None, '___sec36'),
|
||||
('Layers', 2, None, '___sec37'),
|
||||
('Weights and biases', 2, None, '___sec38'),
|
||||
('Feed-forward pass', 2, None, '___sec39'),
|
||||
('Matrix multiplications', 2, None, '___sec40'),
|
||||
('Choose cost function and optimizer', 2, None, '___sec41'),
|
||||
('Optimizing the cost function', 2, None, '___sec42'),
|
||||
('Regularization', 2, None, '___sec43'),
|
||||
('Matrix multiplication', 2, None, '___sec44'),
|
||||
('Improving performance', 2, None, '___sec45'),
|
||||
('Full object-oriented implementation', 2, None, '___sec46'),
|
||||
('Evaluate model performance on test data', 2, None, '___sec47'),
|
||||
('Adjust hyperparameters', 2, None, '___sec48'),
|
||||
('Visualization', 2, None, '___sec49'),
|
||||
('scikit-learn implementation', 2, None, '___sec50'),
|
||||
('Visualization', 2, None, '___sec51'),
|
||||
'___sec34'),
|
||||
('Collect and pre-process data', 2, None, '___sec35'),
|
||||
('Train and test datasets', 2, None, '___sec36'),
|
||||
('Define model and architecture', 2, None, '___sec37'),
|
||||
('Layers', 2, None, '___sec38'),
|
||||
('Weights and biases', 2, None, '___sec39'),
|
||||
('Feed-forward pass', 2, None, '___sec40'),
|
||||
('Matrix multiplications', 2, None, '___sec41'),
|
||||
('Choose cost function and optimizer', 2, None, '___sec42'),
|
||||
('Optimizing the cost function', 2, None, '___sec43'),
|
||||
('Regularization', 2, None, '___sec44'),
|
||||
('Matrix multiplication', 2, None, '___sec45'),
|
||||
('Improving performance', 2, None, '___sec46'),
|
||||
('Full object-oriented implementation', 2, None, '___sec47'),
|
||||
('Evaluate model performance on test data', 2, None, '___sec48'),
|
||||
('Adjust hyperparameters', 2, None, '___sec49'),
|
||||
('Visualization', 2, None, '___sec50'),
|
||||
('scikit-learn implementation', 2, None, '___sec51'),
|
||||
('Visualization', 2, None, '___sec52'),
|
||||
('Building neural networks in Tensorflow and Keras',
|
||||
2,
|
||||
None,
|
||||
'___sec52'),
|
||||
('Tensorflow', 2, None, '___sec53'),
|
||||
('Collect and pre-process data', 2, None, '___sec54'),
|
||||
('Using TensorFlow backend', 2, None, '___sec55'),
|
||||
('Optimizing and using gradient descent', 2, None, '___sec56'),
|
||||
('Using Keras', 2, None, '___sec57'),
|
||||
('Which activation function should I use?', 2, None, '___sec58'),
|
||||
'___sec53'),
|
||||
('Tensorflow', 2, None, '___sec54'),
|
||||
('Collect and pre-process data', 2, None, '___sec55'),
|
||||
('Using TensorFlow backend', 2, None, '___sec56'),
|
||||
('Optimizing and using gradient descent', 2, None, '___sec57'),
|
||||
('Using Keras', 2, None, '___sec58'),
|
||||
('Which activation function should I use?', 2, None, '___sec59'),
|
||||
('Is the Logistic activation function (Sigmoid) our choice?',
|
||||
2,
|
||||
None,
|
||||
'___sec59'),
|
||||
('The derivative of the Logistic funtion', 2, None, '___sec60'),
|
||||
('The RELU function family', 2, None, '___sec61'),
|
||||
('Which activation function should we use?', 2, None, '___sec62'),
|
||||
'___sec60'),
|
||||
('The derivative of the Logistic funtion', 2, None, '___sec61'),
|
||||
('The RELU function family', 2, None, '___sec62'),
|
||||
('Which activation function should we use?', 2, None, '___sec63'),
|
||||
('A top-down perspective on Neural networks',
|
||||
2,
|
||||
None,
|
||||
'___sec63'),
|
||||
'___sec64'),
|
||||
('Limitations of supervised learning with deep networks',
|
||||
2,
|
||||
None,
|
||||
'___sec64')]}
|
||||
'___sec65')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -210,38 +211,39 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs031.html#___sec30" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model for classification</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs032.html#___sec31" style="font-size: 80%;"><b>Defining the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs033.html#___sec32" style="font-size: 80%;"><b>Example: binary classification problem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs034.html#___sec33" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Weights and biases</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Regularization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Improving performance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs053.html#___sec52" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Tensorflow</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Using Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs060.html#___sec59" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec60" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs063.html#___sec62" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs034.html#___sec33" style="font-size: 80%;"><b>The Softmax function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Weights and biases</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Regularization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Improving performance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs053.html#___sec52" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Tensorflow</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Using Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs060.html#___sec59" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec60" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs063.html#___sec62" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs066.html#___sec65" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -255,40 +257,34 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0061"></a>
|
||||
<!-- !split -->
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec60" class="anchor">The derivative of the Logistic funtion </h2>
|
||||
<h2 id="___sec60" class="anchor">Is the Logistic activation function (Sigmoid) our choice? </h2>
|
||||
|
||||
<p>
|
||||
Looking at the logistic activation function, when inputs become large
|
||||
(negative or positive), the function saturates at 0 or 1, with a
|
||||
derivative extremely close to 0. Thus when backpropagation kicks in,
|
||||
it has virtually no gradient to propagate back through the network,
|
||||
and what little gradient exists keeps getting diluted as
|
||||
backpropagation progresses down through the top layers, so there is
|
||||
really nothing left for the lower layers.
|
||||
Although this unfortunate behavior has been empirically observed for
|
||||
quite a while (it was one of the reasons why deep neural networks were
|
||||
mostly abandoned for a long time), it is only around 2010 that
|
||||
significant progress was made in understanding it.
|
||||
|
||||
<p>
|
||||
In their paper, Glorot and Bengio propose a way to significantly
|
||||
alleviate this problem. We need the signal to flow properly in both
|
||||
directions: in the forward direction when making predictions, and in
|
||||
the reverse direction when backpropagating gradients. We don’t want
|
||||
the signal to die out, nor do we want it to explode and saturate. For
|
||||
the signal to flow properly, the authors argue that we need the
|
||||
variance of the outputs of each layer to be equal to the variance of
|
||||
its inputs, and we also need the gradients to have equal variance
|
||||
before and after flowing through a layer in the reverse direction.
|
||||
A paper titled <a href="http://proceedings.mlr.press/v9/glorot10a.html" target="_self">Understanding the Difficulty of Training Deep
|
||||
Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio</a> found that
|
||||
the problems with the popular logistic
|
||||
sigmoid activation function and the weight initialization technique
|
||||
that was most popular at the time, namely random initialization using
|
||||
a normal distribution with a mean of 0 and a standard deviation of
|
||||
1.
|
||||
|
||||
<p>
|
||||
One of the insights in the 2010 paper by Glorot and Bengio was that
|
||||
the vanishing/exploding gradients problems were in part due to a poor
|
||||
choice of activation function. Until then most people had assumed that
|
||||
if Nature had chosen to use roughly sigmoid activation functions in
|
||||
biological neurons, they must be an excellent choice. But it turns out
|
||||
that other activation functions behave much better in deep neural
|
||||
networks, in particular the ReLU activation function, mostly because
|
||||
it does not saturate for positive values (and also because it is quite
|
||||
fast to compute).
|
||||
They showed that with this activation function and this
|
||||
initialization scheme, the variance of the outputs of each layer is
|
||||
much greater than the variance of its inputs. Going forward in the
|
||||
network, the variance keeps increasing after each layer until the
|
||||
activation function saturates at the top layers. This is actually made
|
||||
worse by the fact that the logistic function has a mean of 0.5, not 0
|
||||
(the hyperbolic tangent function has a mean of 0 and behaves slightly
|
||||
better than the logistic function in deep networks).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -310,6 +306,7 @@ fast to compute).
|
||||
<li><a href="._NeuralNet-bs063.html">64</a></li>
|
||||
<li><a href="._NeuralNet-bs064.html">65</a></li>
|
||||
<li><a href="._NeuralNet-bs065.html">66</a></li>
|
||||
<li><a href="._NeuralNet-bs066.html">67</a></li>
|
||||
<li><a href="._NeuralNet-bs062.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
Reference in New Issue
Block a user