small update on neural networks
This commit is contained in:
mhjensen
@@ -92,54 +92,55 @@ Automatically generated HTML file from DocOnce source
|
||||
'___sec30'),
|
||||
('Defining the cost function', 2, None, '___sec31'),
|
||||
('Example: binary classification problem', 2, None, '___sec32'),
|
||||
('The Softmax function', 2, None, '___sec33'),
|
||||
('Developing a code for doing neural networks with back '
|
||||
'propagation',
|
||||
2,
|
||||
None,
|
||||
'___sec33'),
|
||||
('Collect and pre-process data', 2, None, '___sec34'),
|
||||
('Train and test datasets', 2, None, '___sec35'),
|
||||
('Define model and architecture', 2, None, '___sec36'),
|
||||
('Layers', 2, None, '___sec37'),
|
||||
('Weights and biases', 2, None, '___sec38'),
|
||||
('Feed-forward pass', 2, None, '___sec39'),
|
||||
('Matrix multiplications', 2, None, '___sec40'),
|
||||
('Choose cost function and optimizer', 2, None, '___sec41'),
|
||||
('Optimizing the cost function', 2, None, '___sec42'),
|
||||
('Regularization', 2, None, '___sec43'),
|
||||
('Matrix multiplication', 2, None, '___sec44'),
|
||||
('Improving performance', 2, None, '___sec45'),
|
||||
('Full object-oriented implementation', 2, None, '___sec46'),
|
||||
('Evaluate model performance on test data', 2, None, '___sec47'),
|
||||
('Adjust hyperparameters', 2, None, '___sec48'),
|
||||
('Visualization', 2, None, '___sec49'),
|
||||
('scikit-learn implementation', 2, None, '___sec50'),
|
||||
('Visualization', 2, None, '___sec51'),
|
||||
'___sec34'),
|
||||
('Collect and pre-process data', 2, None, '___sec35'),
|
||||
('Train and test datasets', 2, None, '___sec36'),
|
||||
('Define model and architecture', 2, None, '___sec37'),
|
||||
('Layers', 2, None, '___sec38'),
|
||||
('Weights and biases', 2, None, '___sec39'),
|
||||
('Feed-forward pass', 2, None, '___sec40'),
|
||||
('Matrix multiplications', 2, None, '___sec41'),
|
||||
('Choose cost function and optimizer', 2, None, '___sec42'),
|
||||
('Optimizing the cost function', 2, None, '___sec43'),
|
||||
('Regularization', 2, None, '___sec44'),
|
||||
('Matrix multiplication', 2, None, '___sec45'),
|
||||
('Improving performance', 2, None, '___sec46'),
|
||||
('Full object-oriented implementation', 2, None, '___sec47'),
|
||||
('Evaluate model performance on test data', 2, None, '___sec48'),
|
||||
('Adjust hyperparameters', 2, None, '___sec49'),
|
||||
('Visualization', 2, None, '___sec50'),
|
||||
('scikit-learn implementation', 2, None, '___sec51'),
|
||||
('Visualization', 2, None, '___sec52'),
|
||||
('Building neural networks in Tensorflow and Keras',
|
||||
2,
|
||||
None,
|
||||
'___sec52'),
|
||||
('Tensorflow', 2, None, '___sec53'),
|
||||
('Collect and pre-process data', 2, None, '___sec54'),
|
||||
('Using TensorFlow backend', 2, None, '___sec55'),
|
||||
('Optimizing and using gradient descent', 2, None, '___sec56'),
|
||||
('Using Keras', 2, None, '___sec57'),
|
||||
('Which activation function should I use?', 2, None, '___sec58'),
|
||||
'___sec53'),
|
||||
('Tensorflow', 2, None, '___sec54'),
|
||||
('Collect and pre-process data', 2, None, '___sec55'),
|
||||
('Using TensorFlow backend', 2, None, '___sec56'),
|
||||
('Optimizing and using gradient descent', 2, None, '___sec57'),
|
||||
('Using Keras', 2, None, '___sec58'),
|
||||
('Which activation function should I use?', 2, None, '___sec59'),
|
||||
('Is the Logistic activation function (Sigmoid) our choice?',
|
||||
2,
|
||||
None,
|
||||
'___sec59'),
|
||||
('The derivative of the Logistic funtion', 2, None, '___sec60'),
|
||||
('The RELU function family', 2, None, '___sec61'),
|
||||
('Which activation function should we use?', 2, None, '___sec62'),
|
||||
'___sec60'),
|
||||
('The derivative of the Logistic funtion', 2, None, '___sec61'),
|
||||
('The RELU function family', 2, None, '___sec62'),
|
||||
('Which activation function should we use?', 2, None, '___sec63'),
|
||||
('A top-down perspective on Neural networks',
|
||||
2,
|
||||
None,
|
||||
'___sec63'),
|
||||
'___sec64'),
|
||||
('Limitations of supervised learning with deep networks',
|
||||
2,
|
||||
None,
|
||||
'___sec64')]}
|
||||
'___sec65')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -210,38 +211,39 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs031.html#___sec30" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model for classification</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs032.html#___sec31" style="font-size: 80%;"><b>Defining the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs033.html#___sec32" style="font-size: 80%;"><b>Example: binary classification problem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs034.html#___sec33" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Weights and biases</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Regularization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Improving performance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs053.html#___sec52" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Tensorflow</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Using Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec59" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs061.html#___sec60" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs063.html#___sec62" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs034.html#___sec33" style="font-size: 80%;"><b>The Softmax function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Weights and biases</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Regularization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Improving performance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs053.html#___sec52" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Tensorflow</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Using Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec59" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs061.html#___sec60" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs063.html#___sec62" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs066.html#___sec65" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -257,32 +259,31 @@ MathJax.Hub.Config({
|
||||
<a name="part0060"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec59" class="anchor">Is the Logistic activation function (Sigmoid) our choice? </h2>
|
||||
<h2 id="___sec59" class="anchor">Which activation function should I use? </h2>
|
||||
|
||||
<p>
|
||||
Although this unfortunate behavior has been empirically observed for
|
||||
quite a while (it was one of the reasons why deep neural networks were
|
||||
mostly abandoned for a long time), it is only around 2010 that
|
||||
significant progress was made in understanding it.
|
||||
The Back propagation algorithm we derived above works by going from
|
||||
the output layer to the input layer, propagating the error gradient on
|
||||
the way. Once the algorithm has computed the gradient of the cost
|
||||
function with regards to each parameter in the network, it uses these
|
||||
gradients to update each parameter with a Gradient Descent (GD) step.
|
||||
|
||||
<p>
|
||||
A paper titled <a href="http://proceedings.mlr.press/v9/glorot10a.html" target="_self">Understanding the Difficulty of Training Deep
|
||||
Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio</a> found that
|
||||
the problems with the popular logistic
|
||||
sigmoid activation function and the weight initialization technique
|
||||
that was most popular at the time, namely random initialization using
|
||||
a normal distribution with a mean of 0 and a standard deviation of
|
||||
1.
|
||||
Unfortunately for us, the gradients often get smaller and smaller as the
|
||||
algorithm progresses down to the first hidden layers. As a result, the
|
||||
GD update leaves the lower layer connection weights
|
||||
virtually unchanged, and training never converges to a good
|
||||
solution. This is known in the literature as
|
||||
<b>the vanishing gradients problem</b>.
|
||||
|
||||
<p>
|
||||
They showed that with this activation function and this
|
||||
initialization scheme, the variance of the outputs of each layer is
|
||||
much greater than the variance of its inputs. Going forward in the
|
||||
network, the variance keeps increasing after each layer until the
|
||||
activation function saturates at the top layers. This is actually made
|
||||
worse by the fact that the logistic function has a mean of 0.5, not 0
|
||||
(the hyperbolic tangent function has a mean of 0 and behaves slightly
|
||||
better than the logistic function in deep networks).
|
||||
In other cases, the opposite can happen, namely the the gradients can grow bigger and
|
||||
bigger. The result is that many of the layers get large updates of the
|
||||
weights the
|
||||
algorithm diverges. This is the <b>exploding gradients problem</b>, which is
|
||||
mostly encountered in recurrent neural networks. More generally, deep
|
||||
neural networks suffer from unstable gradients, different layers may
|
||||
learn at widely different speeds
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -305,6 +306,7 @@ better than the logistic function in deep networks).
|
||||
<li><a href="._NeuralNet-bs063.html">64</a></li>
|
||||
<li><a href="._NeuralNet-bs064.html">65</a></li>
|
||||
<li><a href="._NeuralNet-bs065.html">66</a></li>
|
||||
<li><a href="._NeuralNet-bs066.html">67</a></li>
|
||||
<li><a href="._NeuralNet-bs061.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
Reference in New Issue
Block a user