small update on neural networks
This commit is contained in:
mhjensen
@@ -92,54 +92,55 @@ Automatically generated HTML file from DocOnce source
|
||||
'___sec30'),
|
||||
('Defining the cost function', 2, None, '___sec31'),
|
||||
('Example: binary classification problem', 2, None, '___sec32'),
|
||||
('The Softmax function', 2, None, '___sec33'),
|
||||
('Developing a code for doing neural networks with back '
|
||||
'propagation',
|
||||
2,
|
||||
None,
|
||||
'___sec33'),
|
||||
('Collect and pre-process data', 2, None, '___sec34'),
|
||||
('Train and test datasets', 2, None, '___sec35'),
|
||||
('Define model and architecture', 2, None, '___sec36'),
|
||||
('Layers', 2, None, '___sec37'),
|
||||
('Weights and biases', 2, None, '___sec38'),
|
||||
('Feed-forward pass', 2, None, '___sec39'),
|
||||
('Matrix multiplications', 2, None, '___sec40'),
|
||||
('Choose cost function and optimizer', 2, None, '___sec41'),
|
||||
('Optimizing the cost function', 2, None, '___sec42'),
|
||||
('Regularization', 2, None, '___sec43'),
|
||||
('Matrix multiplication', 2, None, '___sec44'),
|
||||
('Improving performance', 2, None, '___sec45'),
|
||||
('Full object-oriented implementation', 2, None, '___sec46'),
|
||||
('Evaluate model performance on test data', 2, None, '___sec47'),
|
||||
('Adjust hyperparameters', 2, None, '___sec48'),
|
||||
('Visualization', 2, None, '___sec49'),
|
||||
('scikit-learn implementation', 2, None, '___sec50'),
|
||||
('Visualization', 2, None, '___sec51'),
|
||||
'___sec34'),
|
||||
('Collect and pre-process data', 2, None, '___sec35'),
|
||||
('Train and test datasets', 2, None, '___sec36'),
|
||||
('Define model and architecture', 2, None, '___sec37'),
|
||||
('Layers', 2, None, '___sec38'),
|
||||
('Weights and biases', 2, None, '___sec39'),
|
||||
('Feed-forward pass', 2, None, '___sec40'),
|
||||
('Matrix multiplications', 2, None, '___sec41'),
|
||||
('Choose cost function and optimizer', 2, None, '___sec42'),
|
||||
('Optimizing the cost function', 2, None, '___sec43'),
|
||||
('Regularization', 2, None, '___sec44'),
|
||||
('Matrix multiplication', 2, None, '___sec45'),
|
||||
('Improving performance', 2, None, '___sec46'),
|
||||
('Full object-oriented implementation', 2, None, '___sec47'),
|
||||
('Evaluate model performance on test data', 2, None, '___sec48'),
|
||||
('Adjust hyperparameters', 2, None, '___sec49'),
|
||||
('Visualization', 2, None, '___sec50'),
|
||||
('scikit-learn implementation', 2, None, '___sec51'),
|
||||
('Visualization', 2, None, '___sec52'),
|
||||
('Building neural networks in Tensorflow and Keras',
|
||||
2,
|
||||
None,
|
||||
'___sec52'),
|
||||
('Tensorflow', 2, None, '___sec53'),
|
||||
('Collect and pre-process data', 2, None, '___sec54'),
|
||||
('Using TensorFlow backend', 2, None, '___sec55'),
|
||||
('Optimizing and using gradient descent', 2, None, '___sec56'),
|
||||
('Using Keras', 2, None, '___sec57'),
|
||||
('Which activation function should I use?', 2, None, '___sec58'),
|
||||
'___sec53'),
|
||||
('Tensorflow', 2, None, '___sec54'),
|
||||
('Collect and pre-process data', 2, None, '___sec55'),
|
||||
('Using TensorFlow backend', 2, None, '___sec56'),
|
||||
('Optimizing and using gradient descent', 2, None, '___sec57'),
|
||||
('Using Keras', 2, None, '___sec58'),
|
||||
('Which activation function should I use?', 2, None, '___sec59'),
|
||||
('Is the Logistic activation function (Sigmoid) our choice?',
|
||||
2,
|
||||
None,
|
||||
'___sec59'),
|
||||
('The derivative of the Logistic funtion', 2, None, '___sec60'),
|
||||
('The RELU function family', 2, None, '___sec61'),
|
||||
('Which activation function should we use?', 2, None, '___sec62'),
|
||||
'___sec60'),
|
||||
('The derivative of the Logistic funtion', 2, None, '___sec61'),
|
||||
('The RELU function family', 2, None, '___sec62'),
|
||||
('Which activation function should we use?', 2, None, '___sec63'),
|
||||
('A top-down perspective on Neural networks',
|
||||
2,
|
||||
None,
|
||||
'___sec63'),
|
||||
'___sec64'),
|
||||
('Limitations of supervised learning with deep networks',
|
||||
2,
|
||||
None,
|
||||
'___sec64')]}
|
||||
'___sec65')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -210,38 +211,39 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs031.html#___sec30" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model for classification</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs032.html#___sec31" style="font-size: 80%;"><b>Defining the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs033.html#___sec32" style="font-size: 80%;"><b>Example: binary classification problem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs034.html#___sec33" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Weights and biases</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Regularization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Improving performance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs053.html#___sec52" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Tensorflow</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Using Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs060.html#___sec59" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs061.html#___sec60" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec62" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs034.html#___sec33" style="font-size: 80%;"><b>The Softmax function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Weights and biases</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Regularization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Improving performance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs053.html#___sec52" style="font-size: 80%;"><b>Visualization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Tensorflow</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Using Keras</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs060.html#___sec59" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs061.html#___sec60" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec62" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._NeuralNet-bs066.html#___sec65" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -257,22 +259,29 @@ MathJax.Hub.Config({
|
||||
<a name="part0063"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec62" class="anchor">Which activation function should we use? </h2>
|
||||
<h2 id="___sec62" class="anchor">The RELU function family </h2>
|
||||
|
||||
<p>
|
||||
In general it seems that the ELU activation function is better than
|
||||
the leaky ReLU function (and its variants), which is better than
|
||||
ReLU. ReLU performs better than \( \tanh \) which in turn performs better
|
||||
than the logistic function.
|
||||
The ReLU activation function suffers from a problem known as the dying
|
||||
ReLUs: during training, some neurons effectively die, meaning they
|
||||
stop outputting anything other than 0.
|
||||
|
||||
<p>
|
||||
If runtime
|
||||
performance is an issue, then you may opt for the leaky ReLU function over the
|
||||
ELU function If you don’t
|
||||
want to tweak yet another hyperparameter, you may just use the default
|
||||
\( \alpha \) of \( 0.01 \) for the leaky ReLU, and \( 1 \) for ELU. If you have
|
||||
spare time and computing power, you can use cross-validation or
|
||||
bootstrap to evaluate other activation functions.
|
||||
In some cases, you may find that half of your network’s neurons are
|
||||
dead, especially if you used a large learning rate. During training,
|
||||
if a neuron’s weights get updated such that the weighted sum of the
|
||||
neuron’s inputs is negative, it will start outputting 0. When this
|
||||
happen, the neuron is unlikely to come back to life since the gradient
|
||||
of the ReLU function is 0 when its input is negative.
|
||||
|
||||
<p>
|
||||
To solve this problem, nowadays practitioners use a variant of the ReLU
|
||||
function, such as the leaky ReLU discussed above or the so-called
|
||||
exponential linear unit (ELU) function
|
||||
|
||||
$$
|
||||
ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right.
|
||||
$$
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -292,6 +301,7 @@ bootstrap to evaluate other activation functions.
|
||||
<li class="active"><a href="._NeuralNet-bs063.html">64</a></li>
|
||||
<li><a href="._NeuralNet-bs064.html">65</a></li>
|
||||
<li><a href="._NeuralNet-bs065.html">66</a></li>
|
||||
<li><a href="._NeuralNet-bs066.html">67</a></li>
|
||||
<li><a href="._NeuralNet-bs064.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
Reference in New Issue
Block a user