update
This commit is contained in:
Morten Hjorth-Jensen
@@ -81,10 +81,10 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'simple-neural-network-and-the-back-propagation-equations'),
|
||||
('Layout of a simple neural network with two input nodes, one '
|
||||
'hidden layer and one output node',
|
||||
'hidden layer with two hidden noeds and one output node',
|
||||
2,
|
||||
None,
|
||||
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node'),
|
||||
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node'),
|
||||
('The ouput layer', 2, None, 'the-ouput-layer'),
|
||||
('Compact expressions', 2, None, 'compact-expressions'),
|
||||
('Output layer', 2, None, 'output-layer'),
|
||||
@@ -149,10 +149,15 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'using-the-chain-rule-and-summing-over-all-k-entries'),
|
||||
('Setting up the back propagation algorithm',
|
||||
('Setting up the back propagation algorithm and algorithm for a '
|
||||
'feed forward NN, initalizations',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm'),
|
||||
'setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations'),
|
||||
('Setting up the back propagation algorithm, part 1',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm-part-1'),
|
||||
('Setting up the back propagation algorithm, part 2',
|
||||
2,
|
||||
None,
|
||||
@@ -162,17 +167,12 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm-part-3'),
|
||||
('Updating the gradients', 2, None, 'updating-the-gradients'),
|
||||
('Activation functions', 3, None, 'activation-functions'),
|
||||
('Activation functions', 2, None, 'activation-functions'),
|
||||
('Activation functions, Logistic and Hyperbolic ones',
|
||||
3,
|
||||
None,
|
||||
'activation-functions-logistic-and-hyperbolic-ones'),
|
||||
('Relevance', 3, None, 'relevance'),
|
||||
('Fine-tuning neural network hyperparameters',
|
||||
2,
|
||||
None,
|
||||
'fine-tuning-neural-network-hyperparameters'),
|
||||
('Hidden layers', 2, None, 'hidden-layers'),
|
||||
('Relevance', 2, None, 'relevance'),
|
||||
('Vanishing gradients', 2, None, 'vanishing-gradients'),
|
||||
('Exploding gradients', 2, None, 'exploding-gradients'),
|
||||
('Is the Logistic activation function (Sigmoid) our choice?',
|
||||
@@ -201,6 +201,11 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
2,
|
||||
None,
|
||||
'more-on-activation-functions-output-layers'),
|
||||
('Fine-tuning neural network hyperparameters',
|
||||
2,
|
||||
None,
|
||||
'fine-tuning-neural-network-hyperparameters'),
|
||||
('Hidden layers', 2, None, 'hidden-layers'),
|
||||
('Batch Normalization', 2, None, 'batch-normalization'),
|
||||
('Dropout', 2, None, 'dropout'),
|
||||
('Gradient Clipping', 2, None, 'gradient-clipping'),
|
||||
@@ -219,10 +224,6 @@ doconce format html week42.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
('Limitations of NNs', 2, None, 'limitations-of-nns'),
|
||||
('Homogeneous data', 2, None, 'homogeneous-data'),
|
||||
('More limitations', 2, None, 'more-limitations'),
|
||||
('Setting up the back-propagation algorithm',
|
||||
2,
|
||||
None,
|
||||
'setting-up-the-back-propagation-algorithm'),
|
||||
('Setting up a Multi-layer perceptron model for classification',
|
||||
2,
|
||||
None,
|
||||
@@ -364,7 +365,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs014.html#the-training" style="font-size: 80%;"><b>The training</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs015.html#code-example" style="font-size: 80%;"><b>Code example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs016.html#simple-neural-network-and-the-back-propagation-equations" style="font-size: 80%;"><b>Simple neural network and the back propagation equations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs017.html#layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node" style="font-size: 80%;"><b>Layout of a simple neural network with two input nodes, one hidden layer and one output node</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs017.html#layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node" style="font-size: 80%;"><b>Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs018.html#the-ouput-layer" style="font-size: 80%;"><b>The ouput layer</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs019.html#compact-expressions" style="font-size: 80%;"><b>Compact expressions</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs020.html#output-layer" style="font-size: 80%;"><b>Output layer</b></a></li>
|
||||
@@ -388,35 +389,35 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs038.html#bringing-it-together" style="font-size: 80%;"><b>Bringing it together</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs039.html#final-back-propagating-equation" style="font-size: 80%;"><b>Final back propagating equation</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs040.html#using-the-chain-rule-and-summing-over-all-k-entries" style="font-size: 80%;"><b>Using the chain rule and summing over all \( k \) entries</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#setting-up-the-back-propagation-algorithm" style="font-size: 80%;"><b>Setting up the back propagation algorithm</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs042.html#setting-up-the-back-propagation-algorithm-part-2" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 2</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs043.html#setting-up-the-back-propagation-algorithm-part-3" style="font-size: 80%;"><b>Setting up the Back propagation algorithm, part 3</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs044.html#updating-the-gradients" style="font-size: 80%;"><b>Updating the gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs098.html#activation-functions" style="font-size: 80%;"> Activation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs046.html#activation-functions-logistic-and-hyperbolic-ones" style="font-size: 80%;"> Activation functions, Logistic and Hyperbolic ones</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs047.html#relevance" style="font-size: 80%;"> Relevance</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs048.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;"><b>Fine-tuning neural network hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs049.html#hidden-layers" style="font-size: 80%;"><b>Hidden layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs050.html#vanishing-gradients" style="font-size: 80%;"><b>Vanishing gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs051.html#exploding-gradients" style="font-size: 80%;"><b>Exploding gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs052.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs053.html#logistic-function-as-the-root-of-problems" style="font-size: 80%;"><b>Logistic function as the root of problems</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs054.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs055.html#insights-from-the-paper-by-glorot-and-bengio" style="font-size: 80%;"><b>Insights from the paper by Glorot and Bengio</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs056.html#the-relu-function-family" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs057.html#elu-function" style="font-size: 80%;"><b>ELU function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs058.html#which-activation-function-should-we-use" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs059.html#more-on-activation-functions-output-layers" style="font-size: 80%;"><b>More on activation functions, output layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs060.html#batch-normalization" style="font-size: 80%;"><b>Batch Normalization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs061.html#dropout" style="font-size: 80%;"><b>Dropout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs062.html#gradient-clipping" style="font-size: 80%;"><b>Gradient Clipping</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs063.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs064.html#more-top-down-perspectives" style="font-size: 80%;"><b>More top-down perspectives</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs065.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs066.html#limitations-of-nns" style="font-size: 80%;"><b>Limitations of NNs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#homogeneous-data" style="font-size: 80%;"><b>Homogeneous data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs068.html#more-limitations" style="font-size: 80%;"><b>More limitations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#setting-up-the-back-propagation-algorithm" style="font-size: 80%;"><b>Setting up the back-propagation algorithm</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs041.html#setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations" style="font-size: 80%;"><b>Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs042.html#setting-up-the-back-propagation-algorithm-part-1" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 1</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs043.html#setting-up-the-back-propagation-algorithm-part-2" style="font-size: 80%;"><b>Setting up the back propagation algorithm, part 2</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs044.html#setting-up-the-back-propagation-algorithm-part-3" style="font-size: 80%;"><b>Setting up the Back propagation algorithm, part 3</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs045.html#updating-the-gradients" style="font-size: 80%;"><b>Updating the gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs098.html#activation-functions" style="font-size: 80%;"><b>Activation functions</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs047.html#activation-functions-logistic-and-hyperbolic-ones" style="font-size: 80%;"> Activation functions, Logistic and Hyperbolic ones</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs048.html#relevance" style="font-size: 80%;"><b>Relevance</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs049.html#vanishing-gradients" style="font-size: 80%;"><b>Vanishing gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs050.html#exploding-gradients" style="font-size: 80%;"><b>Exploding gradients</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs051.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs052.html#logistic-function-as-the-root-of-problems" style="font-size: 80%;"><b>Logistic function as the root of problems</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs053.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs054.html#insights-from-the-paper-by-glorot-and-bengio" style="font-size: 80%;"><b>Insights from the paper by Glorot and Bengio</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs055.html#the-relu-function-family" style="font-size: 80%;"><b>The RELU function family</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs056.html#elu-function" style="font-size: 80%;"><b>ELU function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs057.html#which-activation-function-should-we-use" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs058.html#more-on-activation-functions-output-layers" style="font-size: 80%;"><b>More on activation functions, output layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs059.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;"><b>Fine-tuning neural network hyperparameters</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs060.html#hidden-layers" style="font-size: 80%;"><b>Hidden layers</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs061.html#batch-normalization" style="font-size: 80%;"><b>Batch Normalization</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs062.html#dropout" style="font-size: 80%;"><b>Dropout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs063.html#gradient-clipping" style="font-size: 80%;"><b>Gradient Clipping</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs064.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs065.html#more-top-down-perspectives" style="font-size: 80%;"><b>More top-down perspectives</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs066.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#limitations-of-nns" style="font-size: 80%;"><b>Limitations of NNs</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs068.html#homogeneous-data" style="font-size: 80%;"><b>Homogeneous data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs069.html#more-limitations" style="font-size: 80%;"><b>More limitations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs070.html#setting-up-a-multi-layer-perceptron-model-for-classification" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model for classification</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs071.html#defining-the-cost-function" style="font-size: 80%;"><b>Defining the cost function</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week42-bs072.html#example-binary-classification-problem" style="font-size: 80%;"><b>Example: binary classification problem</b></a></li>
|
||||
@@ -464,10 +465,13 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
<a name="part0067"></a>
|
||||
<!-- !split -->
|
||||
<h2 id="homogeneous-data" class="anchor">Homogeneous data </h2>
|
||||
<h2 id="limitations-of-nns" class="anchor">Limitations of NNs </h2>
|
||||
|
||||
<p>Here we list some of the important limitations of supervised neural network based models. </p>
|
||||
|
||||
<ul>
|
||||
<li> <b>Homogeneous data.</b> Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e. some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types.</li>
|
||||
<li> <b>Need labeled data</b>. All supervised learning methods, DNNs for supervised learning require labeled data. Often, labeled data is harder to acquire than unlabeled data (e.g. one must pay for human experts to label images).</li>
|
||||
<li> <b>Supervised neural networks are extremely data intensive.</b> DNNs are data hungry. They perform best when data is plentiful. This is doubly so for supervised methods where the data must also be labeled. The utility of DNNs is extremely limited if data is hard to acquire or the datasets are small (hundreds to a few thousand samples). In this case, the performance of other methods that utilize hand-engineered features can exceed that of DNNs.</li>
|
||||
</ul>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
|
||||
Reference in New Issue
Block a user