lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

week 40

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-10-04 20:57:06 +02:00
parent 623ed51668
commit 9aca260a7f
7 changed files with 776 additions and 708 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week40/html/week40-bs.html
+177 -123
View File
@@ -1,11 +1,11 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 40: From Stochastic Gradient Descent to Neural networks">
@@ -41,90 +41,144 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 40', 2, None, '___sec0'),
('Overview video for week 40', 2, None, '___sec1'),
('Stochastic Gradient Descent', 2, None, '___sec2'),
('Computation of gradients', 2, None, '___sec3'),
('SGD example', 2, None, '___sec4'),
('The gradient step', 2, None, '___sec5'),
('Simple example code', 2, None, '___sec6'),
('When do we stop?', 2, None, '___sec7'),
('Slightly different approach', 2, None, '___sec8'),
('Program for stochastic gradient', 2, None, '___sec9'),
('Momentum based GD', 2, None, '___sec10'),
('More on momentum based approaches', 2, None, '___sec11'),
('Momentum parameter', 2, None, '___sec12'),
('Second moment of the gradient', 2, None, '___sec13'),
('RMS prop', 2, None, '___sec14'),
('ADAM optimizer', 2, None, '___sec15'),
('Practical tips', 2, None, '___sec16'),
('Automatic differentiation', 2, None, '___sec17'),
('Using autograd', 2, None, '___sec18'),
('Autograd with more complicated functions', 2, None, '___sec19'),
'sections': [('Plan for week 40', 2, None, 'plan-for-week-40'),
('Overview video for week 40',
2,
None,
'overview-video-for-week-40'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
('When do we stop?', 2, None, 'when-do-we-stop'),
('Slightly different approach',
2,
None,
'slightly-different-approach'),
('Program for stochastic gradient',
2,
None,
'program-for-stochastic-gradient'),
('Momentum based GD', 2, None, 'momentum-based-gd'),
('More on momentum based approaches',
2,
None,
'more-on-momentum-based-approaches'),
('Momentum parameter', 2, None, 'momentum-parameter'),
('Second moment of the gradient',
2,
None,
'second-moment-of-the-gradient'),
('RMS prop', 2, None, 'rms-prop'),
('ADAM optimizer', 2, None, 'adam-optimizer'),
('Practical tips', 2, None, 'practical-tips'),
('Automatic differentiation',
2,
None,
'automatic-differentiation'),
('Using autograd', 2, None, 'using-autograd'),
('Autograd with more complicated functions',
2,
None,
'autograd-with-more-complicated-functions'),
('More complicated functions using the elements of their '
'arguments directly',
2,
None,
'___sec20'),
'more-complicated-functions-using-the-elements-of-their-arguments-directly'),
('Functions using mathematical functions from Numpy',
2,
None,
'___sec21'),
('More autograd', 2, None, '___sec22'),
('And with loops', 2, None, '___sec23'),
('Using recursion', 2, None, '___sec24'),
('Unsupported functions', 2, None, '___sec25'),
'functions-using-mathematical-functions-from-numpy'),
('More autograd', 2, None, 'more-autograd'),
('And with loops', 2, None, 'and-with-loops'),
('Using recursion', 2, None, 'using-recursion'),
('Unsupported functions', 2, None, 'unsupported-functions'),
('The syntax a.dot(b) when finding the dot product',
2,
None,
'___sec26'),
('Recommended to avoid', 2, None, '___sec27'),
('Neural networks', 2, None, '___sec28'),
('Artificial neurons', 2, None, '___sec29'),
('Neural network types', 2, None, '___sec30'),
('Feed-forward neural networks', 2, None, '___sec31'),
('Convolutional Neural Network', 2, None, '___sec32'),
('Recurrent neural networks', 2, None, '___sec33'),
('Other types of networks', 2, None, '___sec34'),
('Multilayer perceptrons', 2, None, '___sec35'),
('Why multilayer perceptrons?', 2, None, '___sec36'),
('Mathematical model', 2, None, '___sec37'),
('Mathematical model', 2, None, '___sec38'),
('Mathematical model', 2, None, '___sec39'),
('Mathematical model', 2, None, '___sec40'),
('Mathematical model', 2, None, '___sec41'),
('Matrix-vector notation', 3, None, '___sec42'),
('Matrix-vector notation and activation', 3, None, '___sec43'),
('Activation functions', 3, None, '___sec44'),
'the-syntax-a-dot-b-when-finding-the-dot-product'),
('Recommended to avoid', 2, None, 'recommended-to-avoid'),
('Neural networks', 2, None, 'neural-networks'),
('Artificial neurons', 2, None, 'artificial-neurons'),
('Neural network types', 2, None, 'neural-network-types'),
('Feed-forward neural networks',
2,
None,
'feed-forward-neural-networks'),
('Convolutional Neural Network',
2,
None,
'convolutional-neural-network'),
('Recurrent neural networks',
2,
None,
'recurrent-neural-networks'),
('Other types of networks', 2, None, 'other-types-of-networks'),
('Multilayer perceptrons', 2, None, 'multilayer-perceptrons'),
('Why multilayer perceptrons?',
2,
None,
'why-multilayer-perceptrons'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Matrix-vector notation', 3, None, 'matrix-vector-notation'),
('Matrix-vector notation and activation',
3,
None,
'matrix-vector-notation-and-activation'),
('Activation functions', 3, None, 'activation-functions'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'___sec45'),
('Relevance', 3, None, '___sec46'),
('The multilayer perceptron (MLP)', 2, None, '___sec47'),
'activation-functions-logistic-and-hyperbolic-ones'),
('Relevance', 3, None, 'relevance'),
('The multilayer perceptron (MLP)',
2,
None,
'the-multilayer-perceptron-mlp'),
('From one to many layers, the universal approximation theorem',
2,
None,
'___sec48'),
'from-one-to-many-layers-the-universal-approximation-theorem'),
('Deriving the back propagation code for a multilayer perceptron '
'model',
2,
None,
'___sec49'),
('Definitions', 2, None, '___sec50'),
('Derivatives and the chain rule', 2, None, '___sec51'),
('Derivative of the cost function', 2, None, '___sec52'),
'deriving-the-back-propagation-code-for-a-multilayer-perceptron-model'),
('Definitions', 2, None, 'definitions'),
('Derivatives and the chain rule',
2,
None,
'derivatives-and-the-chain-rule'),
('Derivative of the cost function',
2,
None,
'derivative-of-the-cost-function'),
('Bringing it together, first back propagation equation',
2,
None,
'___sec53'),
('Derivatives in terms of $z_j^L$', 2, None, '___sec54'),
('Bringing it together', 2, None, '___sec55'),
('Final back propagating equation', 2, None, '___sec56'),
'bringing-it-together-first-back-propagation-equation'),
('Derivatives in terms of $z_j^L$',
2,
None,
'derivatives-in-terms-of-z_j-l'),
('Bringing it together', 2, None, 'bringing-it-together'),
('Final back propagating equation',
2,
None,
'final-back-propagating-equation'),
('Setting up the Back propagation algorithm',
2,
None,
'___sec57')]}
'setting-up-the-back-propagation-algorithm')]}
end of tocinfo -->
<body>
@@ -162,64 +216,64 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week40-bs001.html#___sec0" style="font-size: 80%;"><b>Plan for week 40</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs002.html#___sec1" style="font-size: 80%;"><b>Overview video for week 40</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs003.html#___sec2" style="font-size: 80%;"><b>Stochastic Gradient Descent</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs004.html#___sec3" style="font-size: 80%;"><b>Computation of gradients</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs005.html#___sec4" style="font-size: 80%;"><b>SGD example</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs006.html#___sec5" style="font-size: 80%;"><b>The gradient step</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs007.html#___sec6" style="font-size: 80%;"><b>Simple example code</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs008.html#___sec7" style="font-size: 80%;"><b>When do we stop?</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs009.html#___sec8" style="font-size: 80%;"><b>Slightly different approach</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs010.html#___sec9" style="font-size: 80%;"><b>Program for stochastic gradient</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs011.html#___sec10" style="font-size: 80%;"><b>Momentum based GD</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs012.html#___sec11" style="font-size: 80%;"><b>More on momentum based approaches</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs013.html#___sec12" style="font-size: 80%;"><b>Momentum parameter</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs014.html#___sec13" style="font-size: 80%;"><b>Second moment of the gradient</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs015.html#___sec14" style="font-size: 80%;"><b>RMS prop</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs016.html#___sec15" style="font-size: 80%;"><b>ADAM optimizer</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs017.html#___sec16" style="font-size: 80%;"><b>Practical tips</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs018.html#___sec17" style="font-size: 80%;"><b>Automatic differentiation</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs019.html#___sec18" style="font-size: 80%;"><b>Using autograd</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs020.html#___sec19" style="font-size: 80%;"><b>Autograd with more complicated functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs021.html#___sec20" style="font-size: 80%;"><b>More complicated functions using the elements of their arguments directly</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs022.html#___sec21" style="font-size: 80%;"><b>Functions using mathematical functions from Numpy</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs023.html#___sec22" style="font-size: 80%;"><b>More autograd</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs024.html#___sec23" style="font-size: 80%;"><b>And with loops</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs025.html#___sec24" style="font-size: 80%;"><b>Using recursion</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs026.html#___sec25" style="font-size: 80%;"><b>Unsupported functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs027.html#___sec26" style="font-size: 80%;"><b>The syntax a.dot(b) when finding the dot product</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs028.html#___sec27" style="font-size: 80%;"><b>Recommended to avoid</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs029.html#___sec28" style="font-size: 80%;"><b>Neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs030.html#___sec29" style="font-size: 80%;"><b>Artificial neurons</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs031.html#___sec30" style="font-size: 80%;"><b>Neural network types</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs032.html#___sec31" style="font-size: 80%;"><b>Feed-forward neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs033.html#___sec32" style="font-size: 80%;"><b>Convolutional Neural Network</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs034.html#___sec33" style="font-size: 80%;"><b>Recurrent neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs035.html#___sec34" style="font-size: 80%;"><b>Other types of networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs036.html#___sec35" style="font-size: 80%;"><b>Multilayer perceptrons</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs037.html#___sec36" style="font-size: 80%;"><b>Why multilayer perceptrons?</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs038.html#___sec37" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs039.html#___sec38" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs040.html#___sec39" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs041.html#___sec40" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs042.html#___sec41" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs043.html#___sec42" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Matrix-vector notation</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs044.html#___sec43" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Matrix-vector notation and activation</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs045.html#___sec44" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Activation functions</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs046.html#___sec45" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Activation functions, Logistic and Hyperbolic ones</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs047.html#___sec46" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Relevance</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs048.html#___sec47" style="font-size: 80%;"><b>The multilayer perceptron (MLP)</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs049.html#___sec48" style="font-size: 80%;"><b>From one to many layers, the universal approximation theorem</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs050.html#___sec49" style="font-size: 80%;"><b>Deriving the back propagation code for a multilayer perceptron model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs051.html#___sec50" style="font-size: 80%;"><b>Definitions</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs052.html#___sec51" style="font-size: 80%;"><b>Derivatives and the chain rule</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs053.html#___sec52" style="font-size: 80%;"><b>Derivative of the cost function</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs054.html#___sec53" style="font-size: 80%;"><b>Bringing it together, first back propagation equation</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs055.html#___sec54" style="font-size: 80%;"><b>Derivatives in terms of \( z_j^L \)</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs056.html#___sec55" style="font-size: 80%;"><b>Bringing it together</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs057.html#___sec56" style="font-size: 80%;"><b>Final back propagating equation</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs058.html#___sec57" style="font-size: 80%;"><b>Setting up the Back propagation algorithm</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs001.html#plan-for-week-40" style="font-size: 80%;"><b>Plan for week 40</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs002.html#overview-video-for-week-40" style="font-size: 80%;"><b>Overview video for week 40</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs003.html#stochastic-gradient-descent" style="font-size: 80%;"><b>Stochastic Gradient Descent</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs004.html#computation-of-gradients" style="font-size: 80%;"><b>Computation of gradients</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs005.html#sgd-example" style="font-size: 80%;"><b>SGD example</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs006.html#the-gradient-step" style="font-size: 80%;"><b>The gradient step</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs007.html#simple-example-code" style="font-size: 80%;"><b>Simple example code</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs008.html#when-do-we-stop" style="font-size: 80%;"><b>When do we stop?</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs009.html#slightly-different-approach" style="font-size: 80%;"><b>Slightly different approach</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs010.html#program-for-stochastic-gradient" style="font-size: 80%;"><b>Program for stochastic gradient</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs011.html#momentum-based-gd" style="font-size: 80%;"><b>Momentum based GD</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs012.html#more-on-momentum-based-approaches" style="font-size: 80%;"><b>More on momentum based approaches</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs013.html#momentum-parameter" style="font-size: 80%;"><b>Momentum parameter</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs014.html#second-moment-of-the-gradient" style="font-size: 80%;"><b>Second moment of the gradient</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs015.html#rms-prop" style="font-size: 80%;"><b>RMS prop</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs016.html#adam-optimizer" style="font-size: 80%;"><b>ADAM optimizer</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs017.html#practical-tips" style="font-size: 80%;"><b>Practical tips</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs018.html#automatic-differentiation" style="font-size: 80%;"><b>Automatic differentiation</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs019.html#using-autograd" style="font-size: 80%;"><b>Using autograd</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs020.html#autograd-with-more-complicated-functions" style="font-size: 80%;"><b>Autograd with more complicated functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs021.html#more-complicated-functions-using-the-elements-of-their-arguments-directly" style="font-size: 80%;"><b>More complicated functions using the elements of their arguments directly</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs022.html#functions-using-mathematical-functions-from-numpy" style="font-size: 80%;"><b>Functions using mathematical functions from Numpy</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs023.html#more-autograd" style="font-size: 80%;"><b>More autograd</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs024.html#and-with-loops" style="font-size: 80%;"><b>And with loops</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs025.html#using-recursion" style="font-size: 80%;"><b>Using recursion</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs026.html#unsupported-functions" style="font-size: 80%;"><b>Unsupported functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs027.html#the-syntax-a-dot-b-when-finding-the-dot-product" style="font-size: 80%;"><b>The syntax a.dot(b) when finding the dot product</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs028.html#recommended-to-avoid" style="font-size: 80%;"><b>Recommended to avoid</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs029.html#neural-networks" style="font-size: 80%;"><b>Neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs030.html#artificial-neurons" style="font-size: 80%;"><b>Artificial neurons</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs031.html#neural-network-types" style="font-size: 80%;"><b>Neural network types</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs032.html#feed-forward-neural-networks" style="font-size: 80%;"><b>Feed-forward neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs033.html#convolutional-neural-network" style="font-size: 80%;"><b>Convolutional Neural Network</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs034.html#recurrent-neural-networks" style="font-size: 80%;"><b>Recurrent neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs035.html#other-types-of-networks" style="font-size: 80%;"><b>Other types of networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs036.html#multilayer-perceptrons" style="font-size: 80%;"><b>Multilayer perceptrons</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs037.html#why-multilayer-perceptrons" style="font-size: 80%;"><b>Why multilayer perceptrons?</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs042.html#mathematical-model" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs042.html#mathematical-model" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs042.html#mathematical-model" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs042.html#mathematical-model" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs042.html#mathematical-model" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs043.html#matrix-vector-notation" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Matrix-vector notation</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs044.html#matrix-vector-notation-and-activation" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Matrix-vector notation and activation</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs045.html#activation-functions" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Activation functions</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs046.html#activation-functions-logistic-and-hyperbolic-ones" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Activation functions, Logistic and Hyperbolic ones</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs047.html#relevance" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Relevance</a></li>
<!-- navigation toc: --> <li><a href="._week40-bs048.html#the-multilayer-perceptron-mlp" style="font-size: 80%;"><b>The multilayer perceptron (MLP)</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs049.html#from-one-to-many-layers-the-universal-approximation-theorem" style="font-size: 80%;"><b>From one to many layers, the universal approximation theorem</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs050.html#deriving-the-back-propagation-code-for-a-multilayer-perceptron-model" style="font-size: 80%;"><b>Deriving the back propagation code for a multilayer perceptron model</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs051.html#definitions" style="font-size: 80%;"><b>Definitions</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs052.html#derivatives-and-the-chain-rule" style="font-size: 80%;"><b>Derivatives and the chain rule</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs053.html#derivative-of-the-cost-function" style="font-size: 80%;"><b>Derivative of the cost function</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs054.html#bringing-it-together-first-back-propagation-equation" style="font-size: 80%;"><b>Bringing it together, first back propagation equation</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs055.html#derivatives-in-terms-of-z_j-l" style="font-size: 80%;"><b>Derivatives in terms of \( z_j^L \)</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs056.html#bringing-it-together" style="font-size: 80%;"><b>Bringing it together</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs057.html#final-back-propagating-equation" style="font-size: 80%;"><b>Final back propagating equation</b></a></li>
<!-- navigation toc: --> <li><a href="._week40-bs058.html#setting-up-the-back-propagation-algorithm" style="font-size: 80%;"><b>Setting up the Back propagation algorithm</b></a></li>
</ul>
</li>
@@ -250,11 +304,11 @@ MathJax.Hub.Config({
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<center>[1] <b>Department of Physics, University of Oslo, Norway</b></center>
<center>[2] <b>Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, USA</b></center>
<br>
<p>
<center><h4>Oct 6, 2020</h4></center> <!-- date -->
<center><h4>Oct 4, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -290,13 +344,13 @@ MathJax.Hub.Config({
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
<a href="https://..."><img width="250" align=right src="https://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week40/html/week40-reveal.html
+89 -88
View File
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 40: From Stochastic Gradient Descent to Neural networks">
@@ -13,7 +13,7 @@
<!-- reveal.js: http://lab.hakim.se/reveal-js/ -->
<!-- reveal.js: https://lab.hakim.se/reveal-js/ -->
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
@@ -76,10 +76,10 @@ document.getElementsByTagName( 'head' )[0].appendChild( link );
.reveal .alert-block > p, .alert-block > ul {margin-bottom:1em}
/*.reveal .alert li {margin-top: 1em}*/
.reveal .alert-block p+p {margin-top:5px}
/*.reveal .alert-notice { background-image: url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_notice.png); }
.reveal .alert-summary { background-image:url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_summary.png); }
.reveal .alert-warning { background-image: url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_warning.png); }
.reveal .alert-question {background-image:url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_question.png); } */
/*.reveal .alert-notice { background-image: url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_notice.png); }
.reveal .alert-summary { background-image:url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_summary.png); }
.reveal .alert-warning { background-image: url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_warning.png); }
.reveal .alert-question {background-image:url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_question.png); } */
</style>
@@ -144,36 +144,37 @@ MathJax.Hub.Config({
<p>&nbsp;<br>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<center>[1] <b>Department of Physics, University of Oslo, Norway</b></center>
<center>[2] <b>Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, USA</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Oct 6, 2020</h4></center> <!-- date -->
<center><h4>Oct 4, 2021</h4></center> <!-- date -->
<br>
<p>
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</section>
<section>
<h2 id="___sec0">Plan for week 40 </h2>
<h2 id="plan-for-week-40">Plan for week 40 </h2>
<ul>
<p><li> Thursday: Stochastic Gradient descent with examples and automatic differentiation and begin Neural Networks. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober1.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<p><li> Friday: Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober2.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<p><li> Thursday: Stochastic Gradient descent with examples and automatic differentiation and begin Neural Networks.</li>
<p><li> Friday: Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.</li>
</ul>
<p>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 10</a> and Hastie et al chapter 11.
For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
For neural networks we recommend Goodfellow et al chapters 6 and 7 and Bishop 5.1-5.4
</section>
<section>
<h2 id="___sec1">Overview video for week 40 </h2>
<h2 id="overview-video-for-week-40">Overview video for week 40 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverviewWeek40.mp4?vrtx=view-as-webpage" target="_blank">Overview Video, from Stochastic Gradient methods to Neural Networks</a>
@@ -181,7 +182,7 @@ For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
<section>
<h2 id="___sec2">Stochastic Gradient Descent </h2>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -201,7 +202,7 @@ $$
<section>
<h2 id="___sec3">Computation of gradients </h2>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>
This in turn means that the gradient can be
@@ -223,7 +224,7 @@ minibatches. We denote these minibatches by \( B_k \) where
<section>
<h2 id="___sec4">SGD example </h2>
<h2 id="sgd-example">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -249,7 +250,7 @@ $$
<section>
<h2 id="___sec5">The gradient step </h2>
<h2 id="the-gradient-step">The gradient step </h2>
<p>
Thus a gradient descent step now looks like
@@ -270,12 +271,12 @@ the number of minibatches, as exemplified in the code below.
<section>
<h2 id="___sec6">Simple example code </h2>
<h2 id="simple-example-code">Simple example code </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
n = <span style="color: #B452CD">100</span> <span style="color: #228B22">#100 datapoints </span>
M = <span style="color: #B452CD">5</span> <span style="color: #228B22">#size of each minibatch</span>
@@ -302,7 +303,7 @@ all \( n \) datapoints.
<section>
<h2 id="___sec7">When do we stop? </h2>
<h2 id="when-do-we-stop">When do we stop? </h2>
<p>
A natural question is when do we stop the search for a new minimum?
@@ -319,7 +320,7 @@ gave the lowest value.
<section>
<h2 id="___sec8">Slightly different approach </h2>
<h2 id="slightly-different-approach">Slightly different approach </h2>
<p>
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -341,7 +342,7 @@ function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">step_length</span>(t,t0,t1):
<span style="color: #8B008B; font-weight: bold">return</span> t0/(t+t1)
@@ -370,12 +371,12 @@ j = <span style="color: #B452CD">0</span>
<section>
<h2 id="___sec9">Program for stochastic gradient </h2>
<h2 id="program-for-stochastic-gradient">Program for stochastic gradient </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">math</span> <span style="color: #8B008B; font-weight: bold">import</span> exp, sqrt
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
@@ -446,7 +447,7 @@ plt.show()
<section>
<h2 id="___sec10">Momentum based GD </h2>
<h2 id="momentum-based-gd">Momentum based GD </h2>
<p>
The stochastic gradient descent (SGD) is almost always used with a
@@ -487,7 +488,7 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
<section>
<h2 id="___sec11">More on momentum based approaches </h2>
<h2 id="more-on-momentum-based-approaches">More on momentum based approaches </h2>
<p>
Let us try to get more intuition from these equations. It is helpful
@@ -523,7 +524,7 @@ $$
<section>
<h2 id="___sec12">Momentum parameter </h2>
<h2 id="momentum-parameter">Momentum parameter </h2>
<p>
Notice that this equation is identical to previous one if we identify
@@ -583,7 +584,7 @@ One of the major advantages of NAG is that it allows for the use of a larger lea
<section>
<h2 id="___sec13">Second moment of the gradient </h2>
<h2 id="second-moment-of-the-gradient">Second moment of the gradient </h2>
<p>
In stochastic gradient descent, with and without momentum, we still
@@ -611,7 +612,7 @@ ADAM.
<section>
<h2 id="___sec14">RMS prop </h2>
<h2 id="rms-prop">RMS prop </h2>
<p>
In RMS prop, in addition to keeping a running average of the first
@@ -644,7 +645,7 @@ learning rate for flat directions.
<section>
<h2 id="___sec15">ADAM optimizer </h2>
<h2 id="adam-optimizer">ADAM optimizer </h2>
<p>
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
@@ -697,7 +698,7 @@ $$
<section>
<h2 id="___sec16">Practical tips </h2>
<h2 id="practical-tips">Practical tips </h2>
<ul>
<p><li> <b>Randomize the data when making mini-batches</b>. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented.</li>
@@ -712,7 +713,7 @@ Geron's text, see chapter 11, has several interesting discussions.
<section>
<h2 id="___sec17">Automatic differentiation </h2>
<h2 id="automatic-differentiation">Automatic differentiation </h2>
<p>
<a href="https://en.wikipedia.org/wiki/Automatic_differentiation" target="_blank">Automatic differentiation (AD)</a>,
@@ -764,7 +765,7 @@ Using <b>autograd</b> we have
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># To do elementwise differentiation:</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> elementwise_grad <span style="color: #8B008B; font-weight: bold">as</span> egrad
@@ -803,7 +804,7 @@ plt.show()
<section>
<h2 id="___sec18">Using autograd </h2>
<h2 id="using-autograd">Using autograd </h2>
<p>
Here we
@@ -815,7 +816,7 @@ experiment with other, possibly more complicated, functions as well.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1</span>(x):
@@ -837,7 +838,7 @@ grad_analytical = <span style="color: #B452CD">3</span>*a**<span style="color: #
<section>
<h2 id="___sec19">Autograd with more complicated functions </h2>
<h2 id="autograd-with-more-complicated-functions">Autograd with more complicated functions </h2>
<p>
To differentiate with respect to two (or more) arguments of a Python
@@ -847,7 +848,7 @@ being differentiated with respect to.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f2</span>(x1,x2):
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">3</span>*x1**<span style="color: #B452CD">3</span> + x2*(x1 - <span style="color: #B452CD">5</span>) + <span style="color: #B452CD">1</span>
@@ -887,12 +888,12 @@ Note that the grad function will not produce the true gradient of the function.
<section>
<h2 id="___sec20">More complicated functions using the elements of their arguments directly </h2>
<h2 id="more-complicated-functions-using-the-elements-of-their-arguments-directly">More complicated functions using the elements of their arguments directly </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f3</span>(x): <span style="color: #228B22"># Assumes x is an array of length 5 or higher</span>
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">2</span>*x[<span style="color: #B452CD">0</span>] + <span style="color: #B452CD">3</span>*x[<span style="color: #B452CD">1</span>] + <span style="color: #B452CD">5</span>*x[<span style="color: #B452CD">2</span>] + <span style="color: #B452CD">7</span>*x[<span style="color: #B452CD">3</span>] + <span style="color: #B452CD">11</span>*x[<span style="color: #B452CD">4</span>]**<span style="color: #B452CD">2</span>
@@ -921,12 +922,12 @@ could expect form a gradient-evaluting function.
<section>
<h2 id="___sec21">Functions using mathematical functions from Numpy </h2>
<h2 id="functions-using-mathematical-functions-from-numpy">Functions using mathematical functions from Numpy </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f4</span>(x):
<span style="color: #8B008B; font-weight: bold">return</span> np.sqrt(<span style="color: #B452CD">1</span>+x**<span style="color: #B452CD">2</span>) + np.exp(x) + np.sin(<span style="color: #B452CD">2</span>*np.pi*x)
@@ -948,12 +949,12 @@ f4_grad_analytical = x/np.sqrt(<span style="color: #B452CD">1</span> + x**<span
<section>
<h2 id="___sec22">More autograd </h2>
<h2 id="more-autograd">More autograd </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f5</span>(x):
<span style="color: #8B008B; font-weight: bold">if</span> x &gt;= <span style="color: #B452CD">0</span>:
@@ -972,12 +973,12 @@ x = <span style="color: #B452CD">2.7</span>
<section>
<h2 id="___sec23">And with loops </h2>
<h2 id="and-with-loops">And with loops </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f6_for</span>(x):
val = <span style="color: #B452CD">0</span>
@@ -1005,7 +1006,7 @@ x = <span style="color: #B452CD">0.5</span>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #228B22"># Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9</span>
<span style="color: #228B22"># The analytical derivative is: sum(i*x**(i-1)) </span>
@@ -1019,11 +1020,11 @@ f6_grad_analytical = <span style="color: #B452CD">0</span>
<section>
<h2 id="___sec24">Using recursion </h2>
<h2 id="using-recursion">Using recursion </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f7</span>(n): <span style="color: #228B22"># Assume that n is an integer</span>
@@ -1057,7 +1058,7 @@ Note that if n is equal to zero or one, Autograd will give an error message. Thi
<section>
<h2 id="___sec25">Unsupported functions </h2>
<h2 id="unsupported-functions">Unsupported functions </h2>
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
<p>
@@ -1065,7 +1066,7 @@ Assigning a value to the variable being differentiated with respect to
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f8</span>(x): <span style="color: #228B22"># Assume x is an array</span>
x[<span style="color: #B452CD">2</span>] = <span style="color: #B452CD">3</span>
@@ -1083,11 +1084,11 @@ Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The
<section>
<h2 id="___sec26">The syntax a.dot(b) when finding the dot product </h2>
<h2 id="the-syntax-a-dot-b-when-finding-the-dot-product">The syntax a.dot(b) when finding the dot product </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f9</span>(a): <span style="color: #228B22"># Assume a is an array with 2 elements</span>
b = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>])
@@ -1107,7 +1108,7 @@ which also computed the dot product can be used:
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f9_alternative</span>(x): <span style="color: #228B22"># Assume a is an array with 2 elements</span>
b = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>])
@@ -1126,12 +1127,12 @@ x = np.array([<span style="color: #B452CD">3.0</span>,<span style="color: #B452C
<section>
<h2 id="___sec27">Recommended to avoid </h2>
<h2 id="recommended-to-avoid">Recommended to avoid </h2>
The documentation recommends to avoid inplace operations such as
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>a += b
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>a += b
a -= b
a*= b
a /=b
@@ -1140,7 +1141,7 @@ a /=b
<section>
<h2 id="___sec28">Neural networks </h2>
<h2 id="neural-networks">Neural networks </h2>
<p>
Artificial neural networks are computational systems that can learn to
@@ -1154,7 +1155,7 @@ a weight variable.
<section>
<h2 id="___sec29">Artificial neurons </h2>
<h2 id="artificial-neurons">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
@@ -1220,7 +1221,7 @@ humanities to life science and medicine.
<section>
<h2 id="___sec30">Neural network types </h2>
<h2 id="neural-network-types">Neural network types </h2>
<p>
An artificial neural network (ANN), is a computational model that
@@ -1247,7 +1248,7 @@ methods we discussed earlier.
<section>
<h2 id="___sec31">Feed-forward neural networks </h2>
<h2 id="feed-forward-neural-networks">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type
@@ -1265,7 +1266,7 @@ to <em>all</em> nodes in the subsequent layer, making this a so-called
<section>
<h2 id="___sec32">Convolutional Neural Network </h2>
<h2 id="convolutional-neural-network">Convolutional Neural Network </h2>
<p>
A different variant of FFNNs are <em>convolutional neural networks</em>
@@ -1291,7 +1292,7 @@ recognition.
<section>
<h2 id="___sec33">Recurrent neural networks </h2>
<h2 id="recurrent-neural-networks">Recurrent neural networks </h2>
<p>
So far we have only mentioned ANNs where information flows in one
@@ -1308,7 +1309,7 @@ especially well-suited for handwriting and speech recognition.
<section>
<h2 id="___sec34">Other types of networks </h2>
<h2 id="other-types-of-networks">Other types of networks </h2>
<p>
There are many other kinds of ANNs that have been developed. One type
@@ -1325,7 +1326,7 @@ type of NN due the unusual activation functions.
<section>
<h2 id="___sec35">Multilayer perceptrons </h2>
<h2 id="multilayer-perceptrons">Multilayer perceptrons </h2>
<p>
One uses often so-called fully-connected feed-forward neural networks
@@ -1339,7 +1340,7 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs).
<section>
<h2 id="___sec36">Why multilayer perceptrons? </h2>
<h2 id="why-multilayer-perceptrons">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward
@@ -1357,7 +1358,7 @@ as to not restrict the range of output values.
<section>
<h2 id="___sec37">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
The output \( y \) is produced via the activation function \( f \)
@@ -1377,7 +1378,7 @@ of the outputs of <em>all</em> neurons in the previous layer.
<section>
<h2 id="___sec38">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( z_i^1 \) of the input coordinates \( x_j \),
@@ -1430,7 +1431,7 @@ is obtained.
<section>
<h2 id="___sec39">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
@@ -1463,7 +1464,7 @@ $$
<section>
<h2 id="___sec40">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden
@@ -1485,7 +1486,7 @@ variables are the input values \( x_n \).
<section>
<h2 id="___sec41">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
This confirms that an MLP, despite its quite convoluted mathematical
@@ -1517,7 +1518,7 @@ flexibility of a neural network.
<section>
<h3 id="___sec42">Matrix-vector notation </h3>
<h3 id="matrix-vector-notation">Matrix-vector notation </h3>
<p>
We can introduce a more convenient notation for the activations in an A NN.
@@ -1559,7 +1560,7 @@ $$
<section>
<h3 id="___sec43">Matrix-vector notation and activation </h3>
<h3 id="matrix-vector-notation-and-activation">Matrix-vector notation and activation </h3>
<p>
The activation of node \( i \) in layer 2 is
@@ -1585,7 +1586,7 @@ used as input to the activation functions. For each operation
<section>
<h3 id="___sec44">Activation functions </h3>
<h3 id="activation-functions">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its
@@ -1608,7 +1609,7 @@ for a FFNN to fulfill the universal approximation theorem
<section>
<h3 id="___sec45">Activation functions, Logistic and Hyperbolic ones </h3>
<h3 id="activation-functions-logistic-and-hyperbolic-ones">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in
@@ -1638,7 +1639,7 @@ $$
<section>
<h3 id="___sec46">Relevance </h3>
<h3 id="relevance">Relevance </h3>
<p>
The <em>sigmoid</em> function are more biologically plausible because the
@@ -1650,7 +1651,7 @@ become the most popular for <em>deep neural networks</em>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #CD5555">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #CD5555">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<span style="color: #CD5555">function that takes any real number, z, and outputs a number (0,1).</span>
<span style="color: #CD5555">It is useful in neural networks for assigning weights on a relative scale.</span>
<span style="color: #CD5555">The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;</span>
@@ -1726,7 +1727,7 @@ plt.show()
<section>
<h2 id="___sec47">The multilayer perceptron (MLP) </h2>
<h2 id="the-multilayer-perceptron-mlp">The multilayer perceptron (MLP) </h2>
<p>
The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of
@@ -1765,7 +1766,7 @@ like logistic regression or linear regression and their modifications on the oth
<section>
<h2 id="___sec48">From one to many layers, the universal approximation theorem </h2>
<h2 id="from-one-to-many-layers-the-universal-approximation-theorem">From one to many layers, the universal approximation theorem </h2>
<p>
A neural network with only one layer, what we called the simple
@@ -1793,7 +1794,7 @@ the potential of being universal approximators.
<section>
<h2 id="___sec49">Deriving the back propagation code for a multilayer perceptron model </h2>
<h2 id="deriving-the-back-propagation-code-for-a-multilayer-perceptron-model">Deriving the back propagation code for a multilayer perceptron model </h2>
<p>
As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
@@ -1826,7 +1827,7 @@ classes.
<section>
<h2 id="___sec50">Definitions </h2>
<h2 id="definitions">Definitions </h2>
<p>
With our definition of the targets \( \hat{t} \), the outputs of the
@@ -1870,7 +1871,7 @@ $$
<section>
<h2 id="___sec51">Derivatives and the chain rule </h2>
<h2 id="derivatives-and-the-chain-rule">Derivatives and the chain rule </h2>
<p>
From the definition of the activation \( z_j^l \) we have
@@ -1898,7 +1899,7 @@ $$
<section>
<h2 id="___sec52">Derivative of the cost function </h2>
<h2 id="derivative-of-the-cost-function">Derivative of the cost function </h2>
<p>
With these definitions we can now compute the derivative of the cost function in terms of the weights.
@@ -1929,7 +1930,7 @@ $$
<section>
<h2 id="___sec53">Bringing it together, first back propagation equation </h2>
<h2 id="bringing-it-together-first-back-propagation-equation">Bringing it together, first back propagation equation </h2>
<p>
We have thus
@@ -1989,7 +1990,7 @@ $$
<section>
<h2 id="___sec54">Derivatives in terms of \( z_j^L \) </h2>
<h2 id="derivatives-in-terms-of-z_j-l">Derivatives in terms of \( z_j^L \) </h2>
<p>
It is also easy to see that our previous equation can be written as
@@ -2012,14 +2013,14 @@ That is, the error \( \delta_j^L \) is exactly equal to the rate of change of th
<section>
<h2 id="___sec55">Bringing it together </h2>
<h2 id="bringing-it-together">Bringing it together </h2>
<p>
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>The starting equations.</b>
<b>The starting equations</b>
<p>&nbsp;<br>
$$
\begin{equation}
@@ -2076,7 +2077,7 @@ one \( L-1 \) in terms of the errors in the final output layer.
<section>
<h2 id="___sec56">Final back propagating equation </h2>
<h2 id="final-back-propagating-equation">Final back propagating equation </h2>
<p>
We have that (replacing \( L \) with a general layer \( l \))
@@ -2116,7 +2117,7 @@ We are now ready to set up the algorithm for back propagation and learning the w
<section>
<h2 id="___sec57">Setting up the Back propagation algorithm </h2>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the Back propagation algorithm </h2>
<p>
The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
doc/pub/week40/html/week40-solarized.html
+203 -148
View File
@@ -1,19 +1,19 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 40: From Stochastic Gradient Descent to Neural networks">
<title>Week 40: From Stochastic Gradient Descent to Neural networks</title>
<link href="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<link href="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<link href="https://thomasf.github.io/solarized-css/solarized-light.min.css" rel="stylesheet">
@@ -48,10 +48,10 @@ pre {
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_question.png); }
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
@@ -61,90 +61,144 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 40', 2, None, '___sec0'),
('Overview video for week 40', 2, None, '___sec1'),
('Stochastic Gradient Descent', 2, None, '___sec2'),
('Computation of gradients', 2, None, '___sec3'),
('SGD example', 2, None, '___sec4'),
('The gradient step', 2, None, '___sec5'),
('Simple example code', 2, None, '___sec6'),
('When do we stop?', 2, None, '___sec7'),
('Slightly different approach', 2, None, '___sec8'),
('Program for stochastic gradient', 2, None, '___sec9'),
('Momentum based GD', 2, None, '___sec10'),
('More on momentum based approaches', 2, None, '___sec11'),
('Momentum parameter', 2, None, '___sec12'),
('Second moment of the gradient', 2, None, '___sec13'),
('RMS prop', 2, None, '___sec14'),
('ADAM optimizer', 2, None, '___sec15'),
('Practical tips', 2, None, '___sec16'),
('Automatic differentiation', 2, None, '___sec17'),
('Using autograd', 2, None, '___sec18'),
('Autograd with more complicated functions', 2, None, '___sec19'),
'sections': [('Plan for week 40', 2, None, 'plan-for-week-40'),
('Overview video for week 40',
2,
None,
'overview-video-for-week-40'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
('When do we stop?', 2, None, 'when-do-we-stop'),
('Slightly different approach',
2,
None,
'slightly-different-approach'),
('Program for stochastic gradient',
2,
None,
'program-for-stochastic-gradient'),
('Momentum based GD', 2, None, 'momentum-based-gd'),
('More on momentum based approaches',
2,
None,
'more-on-momentum-based-approaches'),
('Momentum parameter', 2, None, 'momentum-parameter'),
('Second moment of the gradient',
2,
None,
'second-moment-of-the-gradient'),
('RMS prop', 2, None, 'rms-prop'),
('ADAM optimizer', 2, None, 'adam-optimizer'),
('Practical tips', 2, None, 'practical-tips'),
('Automatic differentiation',
2,
None,
'automatic-differentiation'),
('Using autograd', 2, None, 'using-autograd'),
('Autograd with more complicated functions',
2,
None,
'autograd-with-more-complicated-functions'),
('More complicated functions using the elements of their '
'arguments directly',
2,
None,
'___sec20'),
'more-complicated-functions-using-the-elements-of-their-arguments-directly'),
('Functions using mathematical functions from Numpy',
2,
None,
'___sec21'),
('More autograd', 2, None, '___sec22'),
('And with loops', 2, None, '___sec23'),
('Using recursion', 2, None, '___sec24'),
('Unsupported functions', 2, None, '___sec25'),
'functions-using-mathematical-functions-from-numpy'),
('More autograd', 2, None, 'more-autograd'),
('And with loops', 2, None, 'and-with-loops'),
('Using recursion', 2, None, 'using-recursion'),
('Unsupported functions', 2, None, 'unsupported-functions'),
('The syntax a.dot(b) when finding the dot product',
2,
None,
'___sec26'),
('Recommended to avoid', 2, None, '___sec27'),
('Neural networks', 2, None, '___sec28'),
('Artificial neurons', 2, None, '___sec29'),
('Neural network types', 2, None, '___sec30'),
('Feed-forward neural networks', 2, None, '___sec31'),
('Convolutional Neural Network', 2, None, '___sec32'),
('Recurrent neural networks', 2, None, '___sec33'),
('Other types of networks', 2, None, '___sec34'),
('Multilayer perceptrons', 2, None, '___sec35'),
('Why multilayer perceptrons?', 2, None, '___sec36'),
('Mathematical model', 2, None, '___sec37'),
('Mathematical model', 2, None, '___sec38'),
('Mathematical model', 2, None, '___sec39'),
('Mathematical model', 2, None, '___sec40'),
('Mathematical model', 2, None, '___sec41'),
('Matrix-vector notation', 3, None, '___sec42'),
('Matrix-vector notation and activation', 3, None, '___sec43'),
('Activation functions', 3, None, '___sec44'),
'the-syntax-a-dot-b-when-finding-the-dot-product'),
('Recommended to avoid', 2, None, 'recommended-to-avoid'),
('Neural networks', 2, None, 'neural-networks'),
('Artificial neurons', 2, None, 'artificial-neurons'),
('Neural network types', 2, None, 'neural-network-types'),
('Feed-forward neural networks',
2,
None,
'feed-forward-neural-networks'),
('Convolutional Neural Network',
2,
None,
'convolutional-neural-network'),
('Recurrent neural networks',
2,
None,
'recurrent-neural-networks'),
('Other types of networks', 2, None, 'other-types-of-networks'),
('Multilayer perceptrons', 2, None, 'multilayer-perceptrons'),
('Why multilayer perceptrons?',
2,
None,
'why-multilayer-perceptrons'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Matrix-vector notation', 3, None, 'matrix-vector-notation'),
('Matrix-vector notation and activation',
3,
None,
'matrix-vector-notation-and-activation'),
('Activation functions', 3, None, 'activation-functions'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'___sec45'),
('Relevance', 3, None, '___sec46'),
('The multilayer perceptron (MLP)', 2, None, '___sec47'),
'activation-functions-logistic-and-hyperbolic-ones'),
('Relevance', 3, None, 'relevance'),
('The multilayer perceptron (MLP)',
2,
None,
'the-multilayer-perceptron-mlp'),
('From one to many layers, the universal approximation theorem',
2,
None,
'___sec48'),
'from-one-to-many-layers-the-universal-approximation-theorem'),
('Deriving the back propagation code for a multilayer perceptron '
'model',
2,
None,
'___sec49'),
('Definitions', 2, None, '___sec50'),
('Derivatives and the chain rule', 2, None, '___sec51'),
('Derivative of the cost function', 2, None, '___sec52'),
'deriving-the-back-propagation-code-for-a-multilayer-perceptron-model'),
('Definitions', 2, None, 'definitions'),
('Derivatives and the chain rule',
2,
None,
'derivatives-and-the-chain-rule'),
('Derivative of the cost function',
2,
None,
'derivative-of-the-cost-function'),
('Bringing it together, first back propagation equation',
2,
None,
'___sec53'),
('Derivatives in terms of $z_j^L$', 2, None, '___sec54'),
('Bringing it together', 2, None, '___sec55'),
('Final back propagating equation', 2, None, '___sec56'),
'bringing-it-together-first-back-propagation-equation'),
('Derivatives in terms of $z_j^L$',
2,
None,
'derivatives-in-terms-of-z_j-l'),
('Bringing it together', 2, None, 'bringing-it-together'),
('Final back propagating equation',
2,
None,
'final-back-propagating-equation'),
('Setting up the Back propagation algorithm',
2,
None,
'___sec57')]}
'setting-up-the-back-propagation-algorithm')]}
end of tocinfo -->
<body>
@@ -182,29 +236,30 @@ MathJax.Hub.Config({
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<center>[1] <b>Department of Physics, University of Oslo, Norway</b></center>
<center>[2] <b>Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, USA</b></center>
<br>
<p>
<center><h4>Oct 6, 2020</h4></center> <!-- date -->
<center><h4>Oct 4, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Plan for week 40 </h2>
<h2 id="plan-for-week-40">Plan for week 40 </h2>
<ul>
<li> Thursday: Stochastic Gradient descent with examples and automatic differentiation and begin Neural Networks. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober1.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<li> Friday: Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober2.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<li> Thursday: Stochastic Gradient descent with examples and automatic differentiation and begin Neural Networks.</li>
<li> Friday: Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 10</a> and Hastie et al chapter 11.
For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
For neural networks we recommend Goodfellow et al chapters 6 and 7 and Bishop 5.1-5.4
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Overview video for week 40 </h2>
<h2 id="overview-video-for-week-40">Overview video for week 40 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverviewWeek40.mp4?vrtx=view-as-webpage" target="_blank">Overview Video, from Stochastic Gradient methods to Neural Networks</a>
@@ -212,7 +267,7 @@ For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Stochastic Gradient Descent </h2>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -230,7 +285,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Computation of gradients </h2>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>
This in turn means that the gradient can be
@@ -250,7 +305,7 @@ minibatches. We denote these minibatches by \( B_k \) where
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">SGD example </h2>
<h2 id="sgd-example">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -274,7 +329,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">The gradient step </h2>
<h2 id="the-gradient-step">The gradient step </h2>
<p>
Thus a gradient descent step now looks like
@@ -293,12 +348,12 @@ the number of minibatches, as exemplified in the code below.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Simple example code </h2>
<h2 id="simple-example-code">Simple example code </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
n = <span style="color: #B452CD">100</span> <span style="color: #228B22">#100 datapoints </span>
M = <span style="color: #B452CD">5</span> <span style="color: #228B22">#size of each minibatch</span>
@@ -325,7 +380,7 @@ all \( n \) datapoints.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">When do we stop? </h2>
<h2 id="when-do-we-stop">When do we stop? </h2>
<p>
A natural question is when do we stop the search for a new minimum?
@@ -342,7 +397,7 @@ gave the lowest value.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Slightly different approach </h2>
<h2 id="slightly-different-approach">Slightly different approach </h2>
<p>
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -362,7 +417,7 @@ function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">step_length</span>(t,t0,t1):
<span style="color: #8B008B; font-weight: bold">return</span> t0/(t+t1)
@@ -390,12 +445,12 @@ j = <span style="color: #B452CD">0</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Program for stochastic gradient </h2>
<h2 id="program-for-stochastic-gradient">Program for stochastic gradient </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">math</span> <span style="color: #8B008B; font-weight: bold">import</span> exp, sqrt
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
@@ -466,7 +521,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Momentum based GD </h2>
<h2 id="momentum-based-gd">Momentum based GD </h2>
<p>
The stochastic gradient descent (SGD) is almost always used with a
@@ -503,7 +558,7 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">More on momentum based approaches </h2>
<h2 id="more-on-momentum-based-approaches">More on momentum based approaches </h2>
<p>
Let us try to get more intuition from these equations. It is helpful
@@ -533,7 +588,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Momentum parameter </h2>
<h2 id="momentum-parameter">Momentum parameter </h2>
<p>
Notice that this equation is identical to previous one if we identify
@@ -589,7 +644,7 @@ One of the major advantages of NAG is that it allows for the use of a larger lea
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Second moment of the gradient </h2>
<h2 id="second-moment-of-the-gradient">Second moment of the gradient </h2>
<p>
In stochastic gradient descent, with and without momentum, we still
@@ -617,7 +672,7 @@ ADAM.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">RMS prop </h2>
<h2 id="rms-prop">RMS prop </h2>
<p>
In RMS prop, in addition to keeping a running average of the first
@@ -648,7 +703,7 @@ learning rate for flat directions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">ADAM optimizer </h2>
<h2 id="adam-optimizer">ADAM optimizer </h2>
<p>
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
@@ -697,7 +752,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Practical tips </h2>
<h2 id="practical-tips">Practical tips </h2>
<ul>
<li> <b>Randomize the data when making mini-batches</b>. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented.</li>
@@ -711,7 +766,7 @@ Geron's text, see chapter 11, has several interesting discussions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Automatic differentiation </h2>
<h2 id="automatic-differentiation">Automatic differentiation </h2>
<p>
<a href="https://en.wikipedia.org/wiki/Automatic_differentiation" target="_blank">Automatic differentiation (AD)</a>,
@@ -758,7 +813,7 @@ Using <b>autograd</b> we have
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># To do elementwise differentiation:</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> elementwise_grad <span style="color: #8B008B; font-weight: bold">as</span> egrad
@@ -796,7 +851,7 @@ plt.show()
<p>
<!-- !split -->
<h2 id="___sec18">Using autograd </h2>
<h2 id="using-autograd">Using autograd </h2>
<p>
Here we
@@ -808,7 +863,7 @@ experiment with other, possibly more complicated, functions as well.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1</span>(x):
@@ -829,7 +884,7 @@ grad_analytical = <span style="color: #B452CD">3</span>*a**<span style="color: #
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Autograd with more complicated functions </h2>
<h2 id="autograd-with-more-complicated-functions">Autograd with more complicated functions </h2>
<p>
To differentiate with respect to two (or more) arguments of a Python
@@ -839,7 +894,7 @@ being differentiated with respect to.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f2</span>(x1,x2):
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">3</span>*x1**<span style="color: #B452CD">3</span> + x2*(x1 - <span style="color: #B452CD">5</span>) + <span style="color: #B452CD">1</span>
@@ -879,12 +934,12 @@ Note that the grad function will not produce the true gradient of the function.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">More complicated functions using the elements of their arguments directly </h2>
<h2 id="more-complicated-functions-using-the-elements-of-their-arguments-directly">More complicated functions using the elements of their arguments directly </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f3</span>(x): <span style="color: #228B22"># Assumes x is an array of length 5 or higher</span>
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">2</span>*x[<span style="color: #B452CD">0</span>] + <span style="color: #B452CD">3</span>*x[<span style="color: #B452CD">1</span>] + <span style="color: #B452CD">5</span>*x[<span style="color: #B452CD">2</span>] + <span style="color: #B452CD">7</span>*x[<span style="color: #B452CD">3</span>] + <span style="color: #B452CD">11</span>*x[<span style="color: #B452CD">4</span>]**<span style="color: #B452CD">2</span>
@@ -913,12 +968,12 @@ could expect form a gradient-evaluting function.
<p>
<!-- !split -->
<h2 id="___sec21">Functions using mathematical functions from Numpy </h2>
<h2 id="functions-using-mathematical-functions-from-numpy">Functions using mathematical functions from Numpy </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f4</span>(x):
<span style="color: #8B008B; font-weight: bold">return</span> np.sqrt(<span style="color: #B452CD">1</span>+x**<span style="color: #B452CD">2</span>) + np.exp(x) + np.sin(<span style="color: #B452CD">2</span>*np.pi*x)
@@ -939,12 +994,12 @@ f4_grad_analytical = x/np.sqrt(<span style="color: #B452CD">1</span> + x**<span
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">More autograd </h2>
<h2 id="more-autograd">More autograd </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f5</span>(x):
<span style="color: #8B008B; font-weight: bold">if</span> x &gt;= <span style="color: #B452CD">0</span>:
@@ -962,12 +1017,12 @@ x = <span style="color: #B452CD">2.7</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">And with loops </h2>
<h2 id="and-with-loops">And with loops </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f6_for</span>(x):
val = <span style="color: #B452CD">0</span>
@@ -995,7 +1050,7 @@ x = <span style="color: #B452CD">0.5</span>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #228B22"># Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9</span>
<span style="color: #228B22"># The analytical derivative is: sum(i*x**(i-1)) </span>
@@ -1008,11 +1063,11 @@ f6_grad_analytical = <span style="color: #B452CD">0</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Using recursion </h2>
<h2 id="using-recursion">Using recursion </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f7</span>(n): <span style="color: #228B22"># Assume that n is an integer</span>
@@ -1046,7 +1101,7 @@ Note that if n is equal to zero or one, Autograd will give an error message. Thi
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Unsupported functions </h2>
<h2 id="unsupported-functions">Unsupported functions </h2>
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
<p>
@@ -1054,7 +1109,7 @@ Assigning a value to the variable being differentiated with respect to
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f8</span>(x): <span style="color: #228B22"># Assume x is an array</span>
x[<span style="color: #B452CD">2</span>] = <span style="color: #B452CD">3</span>
@@ -1072,11 +1127,11 @@ Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">The syntax a.dot(b) when finding the dot product </h2>
<h2 id="the-syntax-a-dot-b-when-finding-the-dot-product">The syntax a.dot(b) when finding the dot product </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f9</span>(a): <span style="color: #228B22"># Assume a is an array with 2 elements</span>
b = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>])
@@ -1096,7 +1151,7 @@ which also computed the dot product can be used:
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f9_alternative</span>(x): <span style="color: #228B22"># Assume a is an array with 2 elements</span>
b = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>])
@@ -1114,12 +1169,12 @@ x = np.array([<span style="color: #B452CD">3.0</span>,<span style="color: #B452C
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Recommended to avoid </h2>
<h2 id="recommended-to-avoid">Recommended to avoid </h2>
The documentation recommends to avoid inplace operations such as
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>a += b
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>a += b
a -= b
a*= b
a /=b
@@ -1127,7 +1182,7 @@ a /=b
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">Neural networks </h2>
<h2 id="neural-networks">Neural networks </h2>
<p>
Artificial neural networks are computational systems that can learn to
@@ -1141,7 +1196,7 @@ a weight variable.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">Artificial neurons </h2>
<h2 id="artificial-neurons">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
@@ -1204,7 +1259,7 @@ humanities to life science and medicine.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Neural network types </h2>
<h2 id="neural-network-types">Neural network types </h2>
<p>
An artificial neural network (ANN), is a computational model that
@@ -1231,7 +1286,7 @@ methods we discussed earlier.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Feed-forward neural networks </h2>
<h2 id="feed-forward-neural-networks">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type
@@ -1249,7 +1304,7 @@ to <em>all</em> nodes in the subsequent layer, making this a so-called
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec32">Convolutional Neural Network </h2>
<h2 id="convolutional-neural-network">Convolutional Neural Network </h2>
<p>
A different variant of FFNNs are <em>convolutional neural networks</em>
@@ -1275,7 +1330,7 @@ recognition.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec33">Recurrent neural networks </h2>
<h2 id="recurrent-neural-networks">Recurrent neural networks </h2>
<p>
So far we have only mentioned ANNs where information flows in one
@@ -1292,7 +1347,7 @@ especially well-suited for handwriting and speech recognition.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec34">Other types of networks </h2>
<h2 id="other-types-of-networks">Other types of networks </h2>
<p>
There are many other kinds of ANNs that have been developed. One type
@@ -1309,7 +1364,7 @@ type of NN due the unusual activation functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec35">Multilayer perceptrons </h2>
<h2 id="multilayer-perceptrons">Multilayer perceptrons </h2>
<p>
One uses often so-called fully-connected feed-forward neural networks
@@ -1323,7 +1378,7 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">Why multilayer perceptrons? </h2>
<h2 id="why-multilayer-perceptrons">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward
@@ -1341,7 +1396,7 @@ as to not restrict the range of output values.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
The output \( y \) is produced via the activation function \( f \)
@@ -1359,7 +1414,7 @@ of the outputs of <em>all</em> neurons in the previous layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec38">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( z_i^1 \) of the input coordinates \( x_j \),
@@ -1406,7 +1461,7 @@ is obtained.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
@@ -1435,7 +1490,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden
@@ -1455,7 +1510,7 @@ variables are the input values \( x_n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec41">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
This confirms that an MLP, despite its quite convoluted mathematical
@@ -1484,7 +1539,7 @@ flexibility of a neural network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec42">Matrix-vector notation </h3>
<h3 id="matrix-vector-notation">Matrix-vector notation </h3>
<p>
We can introduce a more convenient notation for the activations in an A NN.
@@ -1523,7 +1578,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec43">Matrix-vector notation and activation </h3>
<h3 id="matrix-vector-notation-and-activation">Matrix-vector notation and activation </h3>
<p>
The activation of node \( i \) in layer 2 is
@@ -1546,7 +1601,7 @@ used as input to the activation functions. For each operation
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec44">Activation functions </h3>
<h3 id="activation-functions">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its
@@ -1563,7 +1618,7 @@ for a FFNN to fulfill the universal approximation theorem
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec45">Activation functions, Logistic and Hyperbolic ones </h3>
<h3 id="activation-functions-logistic-and-hyperbolic-ones">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in
@@ -1588,7 +1643,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec46">Relevance </h3>
<h3 id="relevance">Relevance </h3>
<p>
The <em>sigmoid</em> function are more biologically plausible because the
@@ -1600,7 +1655,7 @@ become the most popular for <em>deep neural networks</em>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #CD5555">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #CD5555">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<span style="color: #CD5555">function that takes any real number, z, and outputs a number (0,1).</span>
<span style="color: #CD5555">It is useful in neural networks for assigning weights on a relative scale.</span>
<span style="color: #CD5555">The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;</span>
@@ -1675,7 +1730,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec47">The multilayer perceptron (MLP) </h2>
<h2 id="the-multilayer-perceptron-mlp">The multilayer perceptron (MLP) </h2>
<p>
The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of
@@ -1713,7 +1768,7 @@ like logistic regression or linear regression and their modifications on the oth
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">From one to many layers, the universal approximation theorem </h2>
<h2 id="from-one-to-many-layers-the-universal-approximation-theorem">From one to many layers, the universal approximation theorem </h2>
<p>
A neural network with only one layer, what we called the simple
@@ -1741,7 +1796,7 @@ the potential of being universal approximators.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">Deriving the back propagation code for a multilayer perceptron model </h2>
<h2 id="deriving-the-back-propagation-code-for-a-multilayer-perceptron-model">Deriving the back propagation code for a multilayer perceptron model </h2>
<p>
As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
@@ -1772,7 +1827,7 @@ classes.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">Definitions </h2>
<h2 id="definitions">Definitions </h2>
<p>
With our definition of the targets \( \hat{t} \), the outputs of the
@@ -1810,7 +1865,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Derivatives and the chain rule </h2>
<h2 id="derivatives-and-the-chain-rule">Derivatives and the chain rule </h2>
<p>
From the definition of the activation \( z_j^l \) we have
@@ -1832,7 +1887,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec52">Derivative of the cost function </h2>
<h2 id="derivative-of-the-cost-function">Derivative of the cost function </h2>
<p>
With these definitions we can now compute the derivative of the cost function in terms of the weights.
@@ -1857,7 +1912,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec53">Bringing it together, first back propagation equation </h2>
<h2 id="bringing-it-together-first-back-propagation-equation">Bringing it together, first back propagation equation </h2>
<p>
We have thus
@@ -1907,7 +1962,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec54">Derivatives in terms of \( z_j^L \) </h2>
<h2 id="derivatives-in-terms-of-z_j-l">Derivatives in terms of \( z_j^L \) </h2>
<p>
It is also easy to see that our previous equation can be written as
@@ -1924,14 +1979,14 @@ $$
That is, the error \( \delta_j^L \) is exactly equal to the rate of change of the cost function as a function of the bias.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec55">Bringing it together </h2>
<h2 id="bringing-it-together">Bringing it together </h2>
<p>
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>The starting equations.</b>
<b>The starting equations</b>
<p>
$$
@@ -1985,7 +2040,7 @@ one \( L-1 \) in terms of the errors in the final output layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec56">Final back propagating equation </h2>
<h2 id="final-back-propagating-equation">Final back propagating equation </h2>
<p>
We have that (replacing \( L \) with a general layer \( l \))
@@ -2017,7 +2072,7 @@ We are now ready to set up the algorithm for back propagation and learning the w
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec57">Setting up the Back propagation algorithm </h2>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the Back propagation algorithm </h2>
<p>
The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
@@ -2091,7 +2146,7 @@ Here it is convenient to use stochastic gradient descent (see the examples below
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week40/html/week40.html
+201 -146
View File
@@ -1,11 +1,11 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 40: From Stochastic Gradient Descent to Neural networks">
@@ -53,10 +53,10 @@ hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_question.png); }
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
@@ -66,90 +66,144 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 40', 2, None, '___sec0'),
('Overview video for week 40', 2, None, '___sec1'),
('Stochastic Gradient Descent', 2, None, '___sec2'),
('Computation of gradients', 2, None, '___sec3'),
('SGD example', 2, None, '___sec4'),
('The gradient step', 2, None, '___sec5'),
('Simple example code', 2, None, '___sec6'),
('When do we stop?', 2, None, '___sec7'),
('Slightly different approach', 2, None, '___sec8'),
('Program for stochastic gradient', 2, None, '___sec9'),
('Momentum based GD', 2, None, '___sec10'),
('More on momentum based approaches', 2, None, '___sec11'),
('Momentum parameter', 2, None, '___sec12'),
('Second moment of the gradient', 2, None, '___sec13'),
('RMS prop', 2, None, '___sec14'),
('ADAM optimizer', 2, None, '___sec15'),
('Practical tips', 2, None, '___sec16'),
('Automatic differentiation', 2, None, '___sec17'),
('Using autograd', 2, None, '___sec18'),
('Autograd with more complicated functions', 2, None, '___sec19'),
'sections': [('Plan for week 40', 2, None, 'plan-for-week-40'),
('Overview video for week 40',
2,
None,
'overview-video-for-week-40'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
('When do we stop?', 2, None, 'when-do-we-stop'),
('Slightly different approach',
2,
None,
'slightly-different-approach'),
('Program for stochastic gradient',
2,
None,
'program-for-stochastic-gradient'),
('Momentum based GD', 2, None, 'momentum-based-gd'),
('More on momentum based approaches',
2,
None,
'more-on-momentum-based-approaches'),
('Momentum parameter', 2, None, 'momentum-parameter'),
('Second moment of the gradient',
2,
None,
'second-moment-of-the-gradient'),
('RMS prop', 2, None, 'rms-prop'),
('ADAM optimizer', 2, None, 'adam-optimizer'),
('Practical tips', 2, None, 'practical-tips'),
('Automatic differentiation',
2,
None,
'automatic-differentiation'),
('Using autograd', 2, None, 'using-autograd'),
('Autograd with more complicated functions',
2,
None,
'autograd-with-more-complicated-functions'),
('More complicated functions using the elements of their '
'arguments directly',
2,
None,
'___sec20'),
'more-complicated-functions-using-the-elements-of-their-arguments-directly'),
('Functions using mathematical functions from Numpy',
2,
None,
'___sec21'),
('More autograd', 2, None, '___sec22'),
('And with loops', 2, None, '___sec23'),
('Using recursion', 2, None, '___sec24'),
('Unsupported functions', 2, None, '___sec25'),
'functions-using-mathematical-functions-from-numpy'),
('More autograd', 2, None, 'more-autograd'),
('And with loops', 2, None, 'and-with-loops'),
('Using recursion', 2, None, 'using-recursion'),
('Unsupported functions', 2, None, 'unsupported-functions'),
('The syntax a.dot(b) when finding the dot product',
2,
None,
'___sec26'),
('Recommended to avoid', 2, None, '___sec27'),
('Neural networks', 2, None, '___sec28'),
('Artificial neurons', 2, None, '___sec29'),
('Neural network types', 2, None, '___sec30'),
('Feed-forward neural networks', 2, None, '___sec31'),
('Convolutional Neural Network', 2, None, '___sec32'),
('Recurrent neural networks', 2, None, '___sec33'),
('Other types of networks', 2, None, '___sec34'),
('Multilayer perceptrons', 2, None, '___sec35'),
('Why multilayer perceptrons?', 2, None, '___sec36'),
('Mathematical model', 2, None, '___sec37'),
('Mathematical model', 2, None, '___sec38'),
('Mathematical model', 2, None, '___sec39'),
('Mathematical model', 2, None, '___sec40'),
('Mathematical model', 2, None, '___sec41'),
('Matrix-vector notation', 3, None, '___sec42'),
('Matrix-vector notation and activation', 3, None, '___sec43'),
('Activation functions', 3, None, '___sec44'),
'the-syntax-a-dot-b-when-finding-the-dot-product'),
('Recommended to avoid', 2, None, 'recommended-to-avoid'),
('Neural networks', 2, None, 'neural-networks'),
('Artificial neurons', 2, None, 'artificial-neurons'),
('Neural network types', 2, None, 'neural-network-types'),
('Feed-forward neural networks',
2,
None,
'feed-forward-neural-networks'),
('Convolutional Neural Network',
2,
None,
'convolutional-neural-network'),
('Recurrent neural networks',
2,
None,
'recurrent-neural-networks'),
('Other types of networks', 2, None, 'other-types-of-networks'),
('Multilayer perceptrons', 2, None, 'multilayer-perceptrons'),
('Why multilayer perceptrons?',
2,
None,
'why-multilayer-perceptrons'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Matrix-vector notation', 3, None, 'matrix-vector-notation'),
('Matrix-vector notation and activation',
3,
None,
'matrix-vector-notation-and-activation'),
('Activation functions', 3, None, 'activation-functions'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'___sec45'),
('Relevance', 3, None, '___sec46'),
('The multilayer perceptron (MLP)', 2, None, '___sec47'),
'activation-functions-logistic-and-hyperbolic-ones'),
('Relevance', 3, None, 'relevance'),
('The multilayer perceptron (MLP)',
2,
None,
'the-multilayer-perceptron-mlp'),
('From one to many layers, the universal approximation theorem',
2,
None,
'___sec48'),
'from-one-to-many-layers-the-universal-approximation-theorem'),
('Deriving the back propagation code for a multilayer perceptron '
'model',
2,
None,
'___sec49'),
('Definitions', 2, None, '___sec50'),
('Derivatives and the chain rule', 2, None, '___sec51'),
('Derivative of the cost function', 2, None, '___sec52'),
'deriving-the-back-propagation-code-for-a-multilayer-perceptron-model'),
('Definitions', 2, None, 'definitions'),
('Derivatives and the chain rule',
2,
None,
'derivatives-and-the-chain-rule'),
('Derivative of the cost function',
2,
None,
'derivative-of-the-cost-function'),
('Bringing it together, first back propagation equation',
2,
None,
'___sec53'),
('Derivatives in terms of $z_j^L$', 2, None, '___sec54'),
('Bringing it together', 2, None, '___sec55'),
('Final back propagating equation', 2, None, '___sec56'),
'bringing-it-together-first-back-propagation-equation'),
('Derivatives in terms of $z_j^L$',
2,
None,
'derivatives-in-terms-of-z_j-l'),
('Bringing it together', 2, None, 'bringing-it-together'),
('Final back propagating equation',
2,
None,
'final-back-propagating-equation'),
('Setting up the Back propagation algorithm',
2,
None,
'___sec57')]}
'setting-up-the-back-propagation-algorithm')]}
end of tocinfo -->
<body>
@@ -187,29 +241,30 @@ MathJax.Hub.Config({
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<center>[1] <b>Department of Physics, University of Oslo, Norway</b></center>
<center>[2] <b>Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, USA</b></center>
<br>
<p>
<center><h4>Oct 6, 2020</h4></center> <!-- date -->
<center><h4>Oct 4, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Plan for week 40 </h2>
<h2 id="plan-for-week-40">Plan for week 40 </h2>
<ul>
<li> Thursday: Stochastic Gradient descent with examples and automatic differentiation and begin Neural Networks. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober1.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<li> Friday: Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober2.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<li> Thursday: Stochastic Gradient descent with examples and automatic differentiation and begin Neural Networks.</li>
<li> Friday: Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 10</a> and Hastie et al chapter 11.
For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
For neural networks we recommend Goodfellow et al chapters 6 and 7 and Bishop 5.1-5.4
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Overview video for week 40 </h2>
<h2 id="overview-video-for-week-40">Overview video for week 40 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverviewWeek40.mp4?vrtx=view-as-webpage" target="_blank">Overview Video, from Stochastic Gradient methods to Neural Networks</a>
@@ -217,7 +272,7 @@ For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Stochastic Gradient Descent </h2>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -235,7 +290,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Computation of gradients </h2>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>
This in turn means that the gradient can be
@@ -255,7 +310,7 @@ minibatches. We denote these minibatches by \( B_k \) where
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">SGD example </h2>
<h2 id="sgd-example">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -279,7 +334,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">The gradient step </h2>
<h2 id="the-gradient-step">The gradient step </h2>
<p>
Thus a gradient descent step now looks like
@@ -298,12 +353,12 @@ the number of minibatches, as exemplified in the code below.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Simple example code </h2>
<h2 id="simple-example-code">Simple example code </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
n <span style="color: #666666">=</span> <span style="color: #666666">100</span> <span style="color: #408080; font-style: italic">#100 datapoints </span>
M <span style="color: #666666">=</span> <span style="color: #666666">5</span> <span style="color: #408080; font-style: italic">#size of each minibatch</span>
@@ -330,7 +385,7 @@ all \( n \) datapoints.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">When do we stop? </h2>
<h2 id="when-do-we-stop">When do we stop? </h2>
<p>
A natural question is when do we stop the search for a new minimum?
@@ -347,7 +402,7 @@ gave the lowest value.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Slightly different approach </h2>
<h2 id="slightly-different-approach">Slightly different approach </h2>
<p>
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -367,7 +422,7 @@ function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">step_length</span>(t,t0,t1):
<span style="color: #008000; font-weight: bold">return</span> t0<span style="color: #666666">/</span>(t<span style="color: #666666">+</span>t1)
@@ -395,12 +450,12 @@ j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Program for stochastic gradient </h2>
<h2 id="program-for-stochastic-gradient">Program for stochastic gradient </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">math</span> <span style="color: #008000; font-weight: bold">import</span> exp, sqrt
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
@@ -471,7 +526,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Momentum based GD </h2>
<h2 id="momentum-based-gd">Momentum based GD </h2>
<p>
The stochastic gradient descent (SGD) is almost always used with a
@@ -508,7 +563,7 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">More on momentum based approaches </h2>
<h2 id="more-on-momentum-based-approaches">More on momentum based approaches </h2>
<p>
Let us try to get more intuition from these equations. It is helpful
@@ -538,7 +593,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Momentum parameter </h2>
<h2 id="momentum-parameter">Momentum parameter </h2>
<p>
Notice that this equation is identical to previous one if we identify
@@ -594,7 +649,7 @@ One of the major advantages of NAG is that it allows for the use of a larger lea
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Second moment of the gradient </h2>
<h2 id="second-moment-of-the-gradient">Second moment of the gradient </h2>
<p>
In stochastic gradient descent, with and without momentum, we still
@@ -622,7 +677,7 @@ ADAM.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">RMS prop </h2>
<h2 id="rms-prop">RMS prop </h2>
<p>
In RMS prop, in addition to keeping a running average of the first
@@ -653,7 +708,7 @@ learning rate for flat directions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">ADAM optimizer </h2>
<h2 id="adam-optimizer">ADAM optimizer </h2>
<p>
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
@@ -702,7 +757,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Practical tips </h2>
<h2 id="practical-tips">Practical tips </h2>
<ul>
<li> <b>Randomize the data when making mini-batches</b>. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented.</li>
@@ -716,7 +771,7 @@ Geron's text, see chapter 11, has several interesting discussions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Automatic differentiation </h2>
<h2 id="automatic-differentiation">Automatic differentiation </h2>
<p>
<a href="https://en.wikipedia.org/wiki/Automatic_differentiation" target="_blank">Automatic differentiation (AD)</a>,
@@ -763,7 +818,7 @@ Using <b>autograd</b> we have
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #408080; font-style: italic"># To do elementwise differentiation:</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> elementwise_grad <span style="color: #008000; font-weight: bold">as</span> egrad
@@ -801,7 +856,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec18">Using autograd </h2>
<h2 id="using-autograd">Using autograd </h2>
<p>
Here we
@@ -813,7 +868,7 @@ experiment with other, possibly more complicated, functions as well.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1</span>(x):
@@ -834,7 +889,7 @@ grad_analytical <span style="color: #666666">=</span> <span style="color: #66666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Autograd with more complicated functions </h2>
<h2 id="autograd-with-more-complicated-functions">Autograd with more complicated functions </h2>
<p>
To differentiate with respect to two (or more) arguments of a Python
@@ -844,7 +899,7 @@ being differentiated with respect to.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f2</span>(x1,x2):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">3*</span>x1<span style="color: #666666">**3</span> <span style="color: #666666">+</span> x2<span style="color: #666666">*</span>(x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>
@@ -884,12 +939,12 @@ Note that the grad function will not produce the true gradient of the function.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">More complicated functions using the elements of their arguments directly </h2>
<h2 id="more-complicated-functions-using-the-elements-of-their-arguments-directly">More complicated functions using the elements of their arguments directly </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f3</span>(x): <span style="color: #408080; font-style: italic"># Assumes x is an array of length 5 or higher</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">2*</span>x[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> <span style="color: #666666">3*</span>x[<span style="color: #666666">1</span>] <span style="color: #666666">+</span> <span style="color: #666666">5*</span>x[<span style="color: #666666">2</span>] <span style="color: #666666">+</span> <span style="color: #666666">7*</span>x[<span style="color: #666666">3</span>] <span style="color: #666666">+</span> <span style="color: #666666">11*</span>x[<span style="color: #666666">4</span>]<span style="color: #666666">**2</span>
@@ -918,12 +973,12 @@ could expect form a gradient-evaluting function.
<p>
<!-- !split -->
<h2 id="___sec21">Functions using mathematical functions from Numpy </h2>
<h2 id="functions-using-mathematical-functions-from-numpy">Functions using mathematical functions from Numpy </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f4</span>(x):
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1+</span>x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>sin(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)
@@ -944,12 +999,12 @@ f4_grad_analytical <span style="color: #666666">=</span> x<span style="color: #6
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">More autograd </h2>
<h2 id="more-autograd">More autograd </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f5</span>(x):
<span style="color: #008000; font-weight: bold">if</span> x <span style="color: #666666">&gt;=</span> <span style="color: #666666">0</span>:
@@ -967,12 +1022,12 @@ x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">And with loops </h2>
<h2 id="and-with-loops">And with loops </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f6_for</span>(x):
val <span style="color: #666666">=</span> <span style="color: #666666">0</span>
@@ -1000,7 +1055,7 @@ x <span style="color: #666666">=</span> <span style="color: #666666">0.5</span>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #408080; font-style: italic"># Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9</span>
<span style="color: #408080; font-style: italic"># The analytical derivative is: sum(i*x**(i-1)) </span>
@@ -1013,11 +1068,11 @@ f6_grad_analytical <span style="color: #666666">=</span> <span style="color: #66
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Using recursion </h2>
<h2 id="using-recursion">Using recursion </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f7</span>(n): <span style="color: #408080; font-style: italic"># Assume that n is an integer</span>
@@ -1051,7 +1106,7 @@ Note that if n is equal to zero or one, Autograd will give an error message. Thi
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Unsupported functions </h2>
<h2 id="unsupported-functions">Unsupported functions </h2>
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
<p>
@@ -1059,7 +1114,7 @@ Assigning a value to the variable being differentiated with respect to
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f8</span>(x): <span style="color: #408080; font-style: italic"># Assume x is an array</span>
x[<span style="color: #666666">2</span>] <span style="color: #666666">=</span> <span style="color: #666666">3</span>
@@ -1077,11 +1132,11 @@ Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">The syntax a.dot(b) when finding the dot product </h2>
<h2 id="the-syntax-a-dot-b-when-finding-the-dot-product">The syntax a.dot(b) when finding the dot product </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f9</span>(a): <span style="color: #408080; font-style: italic"># Assume a is an array with 2 elements</span>
b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">2.0</span>])
@@ -1101,7 +1156,7 @@ which also computed the dot product can be used:
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f9_alternative</span>(x): <span style="color: #408080; font-style: italic"># Assume a is an array with 2 elements</span>
b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">2.0</span>])
@@ -1119,12 +1174,12 @@ x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>a
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Recommended to avoid </h2>
<h2 id="recommended-to-avoid">Recommended to avoid </h2>
The documentation recommends to avoid inplace operations such as
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>a <span style="color: #666666">+=</span> b
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>a <span style="color: #666666">+=</span> b
a <span style="color: #666666">-=</span> b
a<span style="color: #666666">*=</span> b
a <span style="color: #666666">/=</span>b
@@ -1132,7 +1187,7 @@ a <span style="color: #666666">/=</span>b
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">Neural networks </h2>
<h2 id="neural-networks">Neural networks </h2>
<p>
Artificial neural networks are computational systems that can learn to
@@ -1146,7 +1201,7 @@ a weight variable.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">Artificial neurons </h2>
<h2 id="artificial-neurons">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
@@ -1209,7 +1264,7 @@ humanities to life science and medicine.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Neural network types </h2>
<h2 id="neural-network-types">Neural network types </h2>
<p>
An artificial neural network (ANN), is a computational model that
@@ -1236,7 +1291,7 @@ methods we discussed earlier.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Feed-forward neural networks </h2>
<h2 id="feed-forward-neural-networks">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type
@@ -1254,7 +1309,7 @@ to <em>all</em> nodes in the subsequent layer, making this a so-called
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec32">Convolutional Neural Network </h2>
<h2 id="convolutional-neural-network">Convolutional Neural Network </h2>
<p>
A different variant of FFNNs are <em>convolutional neural networks</em>
@@ -1280,7 +1335,7 @@ recognition.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec33">Recurrent neural networks </h2>
<h2 id="recurrent-neural-networks">Recurrent neural networks </h2>
<p>
So far we have only mentioned ANNs where information flows in one
@@ -1297,7 +1352,7 @@ especially well-suited for handwriting and speech recognition.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec34">Other types of networks </h2>
<h2 id="other-types-of-networks">Other types of networks </h2>
<p>
There are many other kinds of ANNs that have been developed. One type
@@ -1314,7 +1369,7 @@ type of NN due the unusual activation functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec35">Multilayer perceptrons </h2>
<h2 id="multilayer-perceptrons">Multilayer perceptrons </h2>
<p>
One uses often so-called fully-connected feed-forward neural networks
@@ -1328,7 +1383,7 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">Why multilayer perceptrons? </h2>
<h2 id="why-multilayer-perceptrons">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward
@@ -1346,7 +1401,7 @@ as to not restrict the range of output values.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
The output \( y \) is produced via the activation function \( f \)
@@ -1364,7 +1419,7 @@ of the outputs of <em>all</em> neurons in the previous layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec38">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( z_i^1 \) of the input coordinates \( x_j \),
@@ -1411,7 +1466,7 @@ is obtained.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
@@ -1440,7 +1495,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden
@@ -1460,7 +1515,7 @@ variables are the input values \( x_n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec41">Mathematical model </h2>
<h2 id="mathematical-model">Mathematical model </h2>
<p>
This confirms that an MLP, despite its quite convoluted mathematical
@@ -1489,7 +1544,7 @@ flexibility of a neural network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec42">Matrix-vector notation </h3>
<h3 id="matrix-vector-notation">Matrix-vector notation </h3>
<p>
We can introduce a more convenient notation for the activations in an A NN.
@@ -1528,7 +1583,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec43">Matrix-vector notation and activation </h3>
<h3 id="matrix-vector-notation-and-activation">Matrix-vector notation and activation </h3>
<p>
The activation of node \( i \) in layer 2 is
@@ -1551,7 +1606,7 @@ used as input to the activation functions. For each operation
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec44">Activation functions </h3>
<h3 id="activation-functions">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its
@@ -1568,7 +1623,7 @@ for a FFNN to fulfill the universal approximation theorem
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec45">Activation functions, Logistic and Hyperbolic ones </h3>
<h3 id="activation-functions-logistic-and-hyperbolic-ones">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in
@@ -1593,7 +1648,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec46">Relevance </h3>
<h3 id="relevance">Relevance </h3>
<p>
The <em>sigmoid</em> function are more biologically plausible because the
@@ -1605,7 +1660,7 @@ become the most popular for <em>deep neural networks</em>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<span style="color: #BA2121; font-style: italic">function that takes any real number, z, and outputs a number (0,1).</span>
<span style="color: #BA2121; font-style: italic">It is useful in neural networks for assigning weights on a relative scale.</span>
<span style="color: #BA2121; font-style: italic">The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;</span>
@@ -1680,7 +1735,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec47">The multilayer perceptron (MLP) </h2>
<h2 id="the-multilayer-perceptron-mlp">The multilayer perceptron (MLP) </h2>
<p>
The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of
@@ -1718,7 +1773,7 @@ like logistic regression or linear regression and their modifications on the oth
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">From one to many layers, the universal approximation theorem </h2>
<h2 id="from-one-to-many-layers-the-universal-approximation-theorem">From one to many layers, the universal approximation theorem </h2>
<p>
A neural network with only one layer, what we called the simple
@@ -1746,7 +1801,7 @@ the potential of being universal approximators.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">Deriving the back propagation code for a multilayer perceptron model </h2>
<h2 id="deriving-the-back-propagation-code-for-a-multilayer-perceptron-model">Deriving the back propagation code for a multilayer perceptron model </h2>
<p>
As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
@@ -1777,7 +1832,7 @@ classes.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">Definitions </h2>
<h2 id="definitions">Definitions </h2>
<p>
With our definition of the targets \( \hat{t} \), the outputs of the
@@ -1815,7 +1870,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Derivatives and the chain rule </h2>
<h2 id="derivatives-and-the-chain-rule">Derivatives and the chain rule </h2>
<p>
From the definition of the activation \( z_j^l \) we have
@@ -1837,7 +1892,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec52">Derivative of the cost function </h2>
<h2 id="derivative-of-the-cost-function">Derivative of the cost function </h2>
<p>
With these definitions we can now compute the derivative of the cost function in terms of the weights.
@@ -1862,7 +1917,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec53">Bringing it together, first back propagation equation </h2>
<h2 id="bringing-it-together-first-back-propagation-equation">Bringing it together, first back propagation equation </h2>
<p>
We have thus
@@ -1912,7 +1967,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec54">Derivatives in terms of \( z_j^L \) </h2>
<h2 id="derivatives-in-terms-of-z_j-l">Derivatives in terms of \( z_j^L \) </h2>
<p>
It is also easy to see that our previous equation can be written as
@@ -1929,14 +1984,14 @@ $$
That is, the error \( \delta_j^L \) is exactly equal to the rate of change of the cost function as a function of the bias.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec55">Bringing it together </h2>
<h2 id="bringing-it-together">Bringing it together </h2>
<p>
We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>The starting equations.</b>
<b>The starting equations</b>
<p>
$$
@@ -1990,7 +2045,7 @@ one \( L-1 \) in terms of the errors in the final output layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec56">Final back propagating equation </h2>
<h2 id="final-back-propagating-equation">Final back propagating equation </h2>
<p>
We have that (replacing \( L \) with a general layer \( l \))
@@ -2022,7 +2077,7 @@ We are now ready to set up the algorithm for back propagation and learning the w
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec57">Setting up the Back propagation algorithm </h2>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the Back propagation algorithm </h2>
<p>
The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
@@ -2096,7 +2151,7 @@ Here it is convenient to use stochastic gradient descent (see the examples below
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week40/ipynb/ipynb-week40-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week40/ipynb/week40.ipynb
+100 -199
View File
File diff suppressed because one or more lines are too long
doc/src/week40/week40.do.txt
+6 -4
View File
@@ -1,16 +1,17 @@
TITLE: Week 40: From Stochastic Gradient Descent to Neural networks
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo, Norway & Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, USA
DATE: today
!split
===== Plan for week 40 =====
* Thursday: Stochastic Gradient descent with examples and automatic differentiation and begin Neural Networks. "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober1.mp4?vrtx=view-as-webpage"
* Friday: Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model. "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober2.mp4?vrtx=view-as-webpage"
* Thursday: Stochastic Gradient descent with examples and automatic differentiation and begin Neural Networks.
* Friday: Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.
Reading suggestions for both days: "Aurelien Geron's chapter 10":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" and Hastie et al chapter 11.
For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
For neural networks we recommend Goodfellow et al chapters 6 and 7 and Bishop 5.1-5.4
!split
===== Overview video for week 40 =====
@@ -1721,3 +1722,4 @@ The parameter $\eta$ is the learning parameter discussed in connection with the
Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.