update
This commit is contained in:
Morten Hjorth-Jensen
@@ -81,10 +81,10 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'simple-neural-network-and-the-back-propagation-equations'),
|
||||
('Layout of a simple neural network with two input nodes, one '
|
||||
'hidden layer and one output node',
|
||||
'hidden layer with two hidden noeds and one output node',
|
||||
2,
|
||||
None,
|
||||
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node'),
|
||||
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node'),
|
||||
('The ouput layer', 2, None, 'the-ouput-layer'),
|
||||
('Compact expressions', 2, None, 'compact-expressions'),
|
||||
('Output layer', 2, None, 'output-layer'),
|
||||
@@ -149,10 +149,15 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'using-the-chain-rule-and-summing-over-all-k-entries'),
|
||||
('Setting up the back propagation algorithm',
|
||||
('Setting up the back propagation algorithm and algorithm for a '
|
||||
'feed forward NN, initalizations',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm'),
|
||||
'setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations'),
|
||||
('Setting up the back propagation algorithm, part 1',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm-part-1'),
|
||||
('Setting up the back propagation algorithm, part 2',
|
||||
2,
|
||||
None,
|
||||
@@ -162,17 +167,12 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm-part-3'),
|
||||
('Updating the gradients', 2, None, 'updating-the-gradients'),
|
||||
('Activation functions', 3, None, 'activation-functions'),
|
||||
('Activation functions', 2, None, 'activation-functions'),
|
||||
('Activation functions, Logistic and Hyperbolic ones',
|
||||
3,
|
||||
None,
|
||||
'activation-functions-logistic-and-hyperbolic-ones'),
|
||||
('Relevance', 3, None, 'relevance'),
|
||||
('Fine-tuning neural network hyperparameters',
|
||||
2,
|
||||
None,
|
||||
'fine-tuning-neural-network-hyperparameters'),
|
||||
('Hidden layers', 2, None, 'hidden-layers'),
|
||||
('Relevance', 2, None, 'relevance'),
|
||||
('Vanishing gradients', 2, None, 'vanishing-gradients'),
|
||||
('Exploding gradients', 2, None, 'exploding-gradients'),
|
||||
('Is the Logistic activation function (Sigmoid) our choice?',
|
||||
@@ -201,6 +201,11 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'more-on-activation-functions-output-layers'),
|
||||
('Fine-tuning neural network hyperparameters',
|
||||
2,
|
||||
None,
|
||||
'fine-tuning-neural-network-hyperparameters'),
|
||||
('Hidden layers', 2, None, 'hidden-layers'),
|
||||
('Batch Normalization', 2, None, 'batch-normalization'),
|
||||
('Dropout', 2, None, 'dropout'),
|
||||
('Gradient Clipping', 2, None, 'gradient-clipping'),
|
||||
@@ -219,10 +224,6 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
('Limitations of NNs', 2, None, 'limitations-of-nns'),
|
||||
('Homogeneous data', 2, None, 'homogeneous-data'),
|
||||
('More limitations', 2, None, 'more-limitations'),
|
||||
('Setting up the back-propagation algorithm',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm'),
|
||||
('Setting up a Multi-layer perceptron model for classification',
|
||||
2,
|
||||
None,
|
||||
@@ -364,7 +365,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs014.html#the-training" style="font-size: 80%;"><b>The training</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs015.html#code-example" style="font-size: 80%;"><b>Code example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs016.html#simple-neural-network-and-the-back-propagation-equations" style="font-size: 80%;"><b>Simple neural network and the back propagation equations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs017.html#layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node" style="font-size: 80%;"><b>Layout of a simple neural network with two input nodes, one hidden layer and one output node</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs017.html#layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node" style="font-size: 80%;"><b>Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs018.html#the-ouput-layer" style="font-size: 80%;"><b>The ouput layer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs019.html#compact-expressions" style="font-size: 80%;"><b>Compact expressions</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs020.html#output-layer" style="font-size: 80%;"><b>Output layer</b></a></li>
|
||||
@@ -388,35 +389,35 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs038.html#bringing-it-together" style="font-size: 80%;"><b>Bringing it together</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs039.html#final-back-propagating-equation" style="font-size: 80%;"><b>Final back propagating equation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs040.html#using-the-chain-rule-and-summing-over-all-k-entries" style="font-size: 80%;"><b>Using the chain rule and summing over all \( k \) entries</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#setting-up-the-back-propagation-algorithm" style="font-size: 80%;"><b>Setting up the back propagation algorithm</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs042.html#setting-up-the-back-propagation-algorithm-part-2" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 2</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs043.html#setting-up-the-back-propagation-algorithm-part-3" style="font-size: 80%;"><b>Setting up the Back propagation algorithm, part 3</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs044.html#updating-the-gradients" style="font-size: 80%;"><b>Updating the gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs098.html#activation-functions" style="font-size: 80%;"> Activation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs046.html#activation-functions-logistic-and-hyperbolic-ones" style="font-size: 80%;"> Activation functions, Logistic and Hyperbolic ones</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs047.html#relevance" style="font-size: 80%;"> Relevance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs048.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;"><b>Fine-tuning neural network hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs049.html#hidden-layers" style="font-size: 80%;"><b>Hidden layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs050.html#vanishing-gradients" style="font-size: 80%;"><b>Vanishing gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs051.html#exploding-gradients" style="font-size: 80%;"><b>Exploding gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs052.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs053.html#logistic-function-as-the-root-of-problems" style="font-size: 80%;"><b>Logistic function as the root of problems</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs054.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs055.html#insights-from-the-paper-by-glorot-and-bengio" style="font-size: 80%;"><b>Insights from the paper by Glorot and Bengio</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs056.html#the-relu-function-family" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs057.html#elu-function" style="font-size: 80%;"><b>ELU function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs058.html#which-activation-function-should-we-use" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs059.html#more-on-activation-functions-output-layers" style="font-size: 80%;"><b>More on activation functions, output layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs060.html#batch-normalization" style="font-size: 80%;"><b>Batch Normalization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs061.html#dropout" style="font-size: 80%;"><b>Dropout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs062.html#gradient-clipping" style="font-size: 80%;"><b>Gradient Clipping</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs063.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#more-top-down-perspectives" style="font-size: 80%;"><b>More top-down perspectives</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs065.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs066.html#limitations-of-nns" style="font-size: 80%;"><b>Limitations of NNs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs067.html#homogeneous-data" style="font-size: 80%;"><b>Homogeneous data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs068.html#more-limitations" style="font-size: 80%;"><b>More limitations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#setting-up-the-back-propagation-algorithm" style="font-size: 80%;"><b>Setting up the back-propagation algorithm</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs041.html#setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations" style="font-size: 80%;"><b>Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs042.html#setting-up-the-back-propagation-algorithm-part-1" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 1</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs043.html#setting-up-the-back-propagation-algorithm-part-2" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 2</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs044.html#setting-up-the-back-propagation-algorithm-part-3" style="font-size: 80%;"><b>Setting up the Back propagation algorithm, part 3</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs045.html#updating-the-gradients" style="font-size: 80%;"><b>Updating the gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs098.html#activation-functions" style="font-size: 80%;"><b>Activation functions</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs047.html#activation-functions-logistic-and-hyperbolic-ones" style="font-size: 80%;"> Activation functions, Logistic and Hyperbolic ones</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs048.html#relevance" style="font-size: 80%;"><b>Relevance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs049.html#vanishing-gradients" style="font-size: 80%;"><b>Vanishing gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs050.html#exploding-gradients" style="font-size: 80%;"><b>Exploding gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs051.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs052.html#logistic-function-as-the-root-of-problems" style="font-size: 80%;"><b>Logistic function as the root of problems</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs053.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs054.html#insights-from-the-paper-by-glorot-and-bengio" style="font-size: 80%;"><b>Insights from the paper by Glorot and Bengio</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs055.html#the-relu-function-family" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs056.html#elu-function" style="font-size: 80%;"><b>ELU function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs057.html#which-activation-function-should-we-use" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs058.html#more-on-activation-functions-output-layers" style="font-size: 80%;"><b>More on activation functions, output layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs059.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;"><b>Fine-tuning neural network hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs060.html#hidden-layers" style="font-size: 80%;"><b>Hidden layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs061.html#batch-normalization" style="font-size: 80%;"><b>Batch Normalization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs062.html#dropout" style="font-size: 80%;"><b>Dropout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs063.html#gradient-clipping" style="font-size: 80%;"><b>Gradient Clipping</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#a-top-down-perspective-on-neural-networks" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs065.html#more-top-down-perspectives" style="font-size: 80%;"><b>More top-down perspectives</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs066.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs067.html#limitations-of-nns" style="font-size: 80%;"><b>Limitations of NNs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs068.html#homogeneous-data" style="font-size: 80%;"><b>Homogeneous data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#more-limitations" style="font-size: 80%;"><b>More limitations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs070.html#setting-up-a-multi-layer-perceptron-model-for-classification" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model for classification</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs071.html#defining-the-cost-function" style="font-size: 80%;"><b>Defining the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs072.html#example-binary-classification-problem" style="font-size: 80%;"><b>Example: binary classification problem</b></a></li>
|
||||
@@ -463,30 +464,25 @@ MathJax.Hub.Config({
|
||||
<div class="container">
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
<a name="part0064"></a>
|
||||
<!-- !split -->
|
||||
<h2 id="more-top-down-perspectives" class="anchor">More top-down perspectives </h2>
|
||||
<!-- !split -->
|
||||
<h2 id="a-top-down-perspective-on-neural-networks" class="anchor">A top-down perspective on Neural networks </h2>
|
||||
|
||||
<p>If the validation and test sets are drawn from the same distributions,
|
||||
then a good performance on the validation set should lead to similarly
|
||||
good performance on the test set.
|
||||
</p>
|
||||
|
||||
<p>However, sometimes
|
||||
the training data and test data differ in subtle ways because, for
|
||||
example, they are collected using slightly different methods, or
|
||||
because it is cheaper to collect data in one way versus another. In
|
||||
this case, there can be a mismatch between the training and test
|
||||
data. This can lead to the neural network overfitting these small
|
||||
differences between the test and training sets, and a poor performance
|
||||
on the test set despite having a good performance on the validation
|
||||
set. To rectify this, Andrew Ng suggests making two validation or dev
|
||||
sets, one constructed from the training data and one constructed from
|
||||
the test data. The difference between the performance of the algorithm
|
||||
on these two validation sets quantifies the train-test mismatch. This
|
||||
can serve as another important diagnostic when using DNNs for
|
||||
supervised learning.
|
||||
<p>The first thing we would like to do is divide the data into two or
|
||||
three parts. A training set, a validation or dev (development) set,
|
||||
and a test set. The test set is the data on which we want to make
|
||||
predictions. The dev set is a subset of the training data we use to
|
||||
check how well we are doing out-of-sample, after training the model on
|
||||
the training dataset. We use the validation error as a proxy for the
|
||||
test error in order to make tweaks to our model. It is crucial that we
|
||||
do not use any of the test data to train the algorithm. This is a
|
||||
cardinal sin in ML. Then:
|
||||
</p>
|
||||
|
||||
<ol>
|
||||
<li> Estimate optimal error rate</li>
|
||||
<li> Minimize underfitting (bias) on training data set.</li>
|
||||
<li> Make sure you are not overfitting.</li>
|
||||
</ol>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
|
||||
Reference in New Issue
Block a user