update
This commit is contained in:
Morten Hjorth-Jensen
@@ -81,10 +81,10 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'simple-neural-network-and-the-back-propagation-equations'),
|
||||
('Layout of a simple neural network with two input nodes, one '
|
||||
'hidden layer and one output node',
|
||||
'hidden layer with two hidden noeds and one output node',
|
||||
2,
|
||||
None,
|
||||
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node'),
|
||||
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node'),
|
||||
('The ouput layer', 2, None, 'the-ouput-layer'),
|
||||
('Compact expressions', 2, None, 'compact-expressions'),
|
||||
('Output layer', 2, None, 'output-layer'),
|
||||
@@ -149,10 +149,15 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'using-the-chain-rule-and-summing-over-all-k-entries'),
|
||||
('Setting up the back propagation algorithm',
|
||||
('Setting up the back propagation algorithm and algorithm for a '
|
||||
'feed forward NN, initalizations',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm'),
|
||||
'setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations'),
|
||||
('Setting up the back propagation algorithm, part 1',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm-part-1'),
|
||||
('Setting up the back propagation algorithm, part 2',
|
||||
2,
|
||||
None,
|
||||
@@ -162,17 +167,12 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm-part-3'),
|
||||
('Updating the gradients', 2, None, 'updating-the-gradients'),
|
||||
('Activation functions', 3, None, 'activation-functions'),
|
||||
('Activation functions', 2, None, 'activation-functions'),
|
||||
('Activation functions, Logistic and Hyperbolic ones',
|
||||
3,
|
||||
None,
|
||||
'activation-functions-logistic-and-hyperbolic-ones'),
|
||||
('Relevance', 3, None, 'relevance'),
|
||||
('Fine-tuning neural network hyperparameters',
|
||||
2,
|
||||
None,
|
||||
'fine-tuning-neural-network-hyperparameters'),
|
||||
('Hidden layers', 2, None, 'hidden-layers'),
|
||||
('Relevance', 2, None, 'relevance'),
|
||||
('Vanishing gradients', 2, None, 'vanishing-gradients'),
|
||||
('Exploding gradients', 2, None, 'exploding-gradients'),
|
||||
('Is the Logistic activation function (Sigmoid) our choice?',
|
||||
@@ -201,6 +201,11 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'more-on-activation-functions-output-layers'),
|
||||
('Fine-tuning neural network hyperparameters',
|
||||
2,
|
||||
None,
|
||||
'fine-tuning-neural-network-hyperparameters'),
|
||||
('Hidden layers', 2, None, 'hidden-layers'),
|
||||
('Batch Normalization', 2, None, 'batch-normalization'),
|
||||
('Dropout', 2, None, 'dropout'),
|
||||
('Gradient Clipping', 2, None, 'gradient-clipping'),
|
||||
@@ -219,10 +224,6 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
('Limitations of NNs', 2, None, 'limitations-of-nns'),
|
||||
('Homogeneous data', 2, None, 'homogeneous-data'),
|
||||
('More limitations', 2, None, 'more-limitations'),
|
||||
('Setting up the back-propagation algorithm',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm'),
|
||||
('Setting up a Multi-layer perceptron model for classification',
|
||||
2,
|
||||
None,
|
||||
@@ -364,7 +365,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs014.html#the-training" style="font-size: 80%;"><b>The training</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs015.html#code-example" style="font-size: 80%;"><b>Code example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs016.html#simple-neural-network-and-the-back-propagation-equations" style="font-size: 80%;"><b>Simple neural network and the back propagation equations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs017.html#layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node" style="font-size: 80%;"><b>Layout of a simple neural network with two input nodes, one hidden layer and one output node</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs017.html#layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node" style="font-size: 80%;"><b>Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs018.html#the-ouput-layer" style="font-size: 80%;"><b>The ouput layer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs019.html#compact-expressions" style="font-size: 80%;"><b>Compact expressions</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs020.html#output-layer" style="font-size: 80%;"><b>Output layer</b></a></li>
|
||||
@@ -388,35 +389,35 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs038.html#bringing-it-together" style="font-size: 80%;"><b>Bringing it together</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs039.html#final-back-propagating-equation" style="font-size: 80%;"><b>Final back propagating equation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs040.html#using-the-chain-rule-and-summing-over-all-k-entries" style="font-size: 80%;"><b>Using the chain rule and summing over all \( k \) entries</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#setting-up-the-back-propagation-algorithm" style="font-size: 80%;"><b>Setting up the back propagation algorithm</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs042.html#setting-up-the-back-propagation-algorithm-part-2" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 2</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs043.html#setting-up-the-back-propagation-algorithm-part-3" style="font-size: 80%;"><b>Setting up the Back propagation algorithm, part 3</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs044.html#updating-the-gradients" style="font-size: 80%;"><b>Updating the gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs098.html#activation-functions" style="font-size: 80%;"> Activation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs046.html#activation-functions-logistic-and-hyperbolic-ones" style="font-size: 80%;"> Activation functions, Logistic and Hyperbolic ones</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs047.html#relevance" style="font-size: 80%;"> Relevance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs048.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;"><b>Fine-tuning neural network hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs049.html#hidden-layers" style="font-size: 80%;"><b>Hidden layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs050.html#vanishing-gradients" style="font-size: 80%;"><b>Vanishing gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs051.html#exploding-gradients" style="font-size: 80%;"><b>Exploding gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs052.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs053.html#logistic-function-as-the-root-of-problems" style="font-size: 80%;"><b>Logistic function as the root of problems</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs054.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs055.html#insights-from-the-paper-by-glorot-and-bengio" style="font-size: 80%;"><b>Insights from the paper by Glorot and Bengio</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs056.html#the-relu-function-family" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs057.html#elu-function" style="font-size: 80%;"><b>ELU function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs058.html#which-activation-function-should-we-use" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs059.html#more-on-activation-functions-output-layers" style="font-size: 80%;"><b>More on activation functions, output layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs060.html#batch-normalization" style="font-size: 80%;"><b>Batch Normalization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs061.html#dropout" style="font-size: 80%;"><b>Dropout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs062.html#gradient-clipping" style="font-size: 80%;"><b>Gradient Clipping</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs063.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs064.html#more-top-down-perspectives" style="font-size: 80%;"><b>More top-down perspectives</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs065.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs066.html#limitations-of-nns" style="font-size: 80%;"><b>Limitations of NNs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs067.html#homogeneous-data" style="font-size: 80%;"><b>Homogeneous data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#more-limitations" style="font-size: 80%;"><b>More limitations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#setting-up-the-back-propagation-algorithm" style="font-size: 80%;"><b>Setting up the back-propagation algorithm</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs041.html#setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations" style="font-size: 80%;"><b>Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs042.html#setting-up-the-back-propagation-algorithm-part-1" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 1</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs043.html#setting-up-the-back-propagation-algorithm-part-2" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 2</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs044.html#setting-up-the-back-propagation-algorithm-part-3" style="font-size: 80%;"><b>Setting up the Back propagation algorithm, part 3</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs045.html#updating-the-gradients" style="font-size: 80%;"><b>Updating the gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs098.html#activation-functions" style="font-size: 80%;"><b>Activation functions</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs047.html#activation-functions-logistic-and-hyperbolic-ones" style="font-size: 80%;"> Activation functions, Logistic and Hyperbolic ones</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs048.html#relevance" style="font-size: 80%;"><b>Relevance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs049.html#vanishing-gradients" style="font-size: 80%;"><b>Vanishing gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs050.html#exploding-gradients" style="font-size: 80%;"><b>Exploding gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs051.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs052.html#logistic-function-as-the-root-of-problems" style="font-size: 80%;"><b>Logistic function as the root of problems</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs053.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs054.html#insights-from-the-paper-by-glorot-and-bengio" style="font-size: 80%;"><b>Insights from the paper by Glorot and Bengio</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs055.html#the-relu-function-family" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs056.html#elu-function" style="font-size: 80%;"><b>ELU function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs057.html#which-activation-function-should-we-use" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs058.html#more-on-activation-functions-output-layers" style="font-size: 80%;"><b>More on activation functions, output layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs059.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;"><b>Fine-tuning neural network hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs060.html#hidden-layers" style="font-size: 80%;"><b>Hidden layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs061.html#batch-normalization" style="font-size: 80%;"><b>Batch Normalization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs062.html#dropout" style="font-size: 80%;"><b>Dropout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs063.html#gradient-clipping" style="font-size: 80%;"><b>Gradient Clipping</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs064.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs065.html#more-top-down-perspectives" style="font-size: 80%;"><b>More top-down perspectives</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs066.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs067.html#limitations-of-nns" style="font-size: 80%;"><b>Limitations of NNs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#homogeneous-data" style="font-size: 80%;"><b>Homogeneous data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#more-limitations" style="font-size: 80%;"><b>More limitations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs070.html#setting-up-a-multi-layer-perceptron-model-for-classification" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model for classification</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs071.html#defining-the-cost-function" style="font-size: 80%;"><b>Defining the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs072.html#example-binary-classification-problem" style="font-size: 80%;"><b>Example: binary classification problem</b></a></li>
|
||||
@@ -464,13 +465,11 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
<a name="part0068"></a>
|
||||
<!-- !split -->
|
||||
<h2 id="more-limitations" class="anchor">More limitations </h2>
|
||||
<h2 id="homogeneous-data" class="anchor">Homogeneous data </h2>
|
||||
|
||||
<ul>
|
||||
<li> <b>Many problems are not about prediction.</b> In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a <em>wrong</em> model. The model might or might not be useful for understanding the underlying science.</li>
|
||||
<li> <b>Homogeneous data.</b> Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e. some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types.</li>
|
||||
</ul>
|
||||
<p>Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems.</p>
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
|
||||
Reference in New Issue
Block a user