Update on descent methods
This commit is contained in:
mhjensen
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -234,7 +266,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._Splines-bs008.html">9</a></li>
|
||||
<li><a href="._Splines-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -220,7 +252,7 @@ some approximative/numerical method to compute the minimum.
|
||||
<li><a href="._Splines-bs009.html">10</a></li>
|
||||
<li><a href="._Splines-bs010.html">11</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs002.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -228,7 +260,7 @@ where \( \hat{\beta} \) are the weights we wish to extract from data, in our cas
|
||||
<li><a href="._Splines-bs010.html">11</a></li>
|
||||
<li><a href="._Splines-bs011.html">12</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs003.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -233,7 +265,7 @@ This defines what we call the Hessian.
|
||||
<li><a href="._Splines-bs011.html">12</a></li>
|
||||
<li><a href="._Splines-bs012.html">13</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs004.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -234,7 +266,7 @@ If we can compute these matrices, in particular the Hessian, the above is often
|
||||
<li><a href="._Splines-bs012.html">13</a></li>
|
||||
<li><a href="._Splines-bs013.html">14</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs005.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -228,7 +260,7 @@ discourage the use of this method.
|
||||
<li><a href="._Splines-bs013.html">14</a></li>
|
||||
<li><a href="._Splines-bs014.html">15</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs006.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -246,7 +278,7 @@ $$
|
||||
<li><a href="._Splines-bs014.html">15</a></li>
|
||||
<li><a href="._Splines-bs015.html">16</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs007.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -229,7 +261,7 @@ vanishes, then Newton-Raphson may fail totally
|
||||
<li><a href="._Splines-bs015.html">16</a></li>
|
||||
<li><a href="._Splines-bs016.html">17</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs008.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -267,7 +299,7 @@ more than two non-linear equations. In our case, the Jacobian matrix is given by
|
||||
<li><a href="._Splines-bs016.html">17</a></li>
|
||||
<li><a href="._Splines-bs017.html">18</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs009.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -237,7 +269,7 @@ we are always moving towards smaller function values, i.e a minimum.
|
||||
<li><a href="._Splines-bs017.html">18</a></li>
|
||||
<li><a href="._Splines-bs018.html">19</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs010.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -233,7 +265,7 @@ the learning rate within the context of Machine Learning.
|
||||
<li><a href="._Splines-bs018.html">19</a></li>
|
||||
<li><a href="._Splines-bs019.html">20</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs011.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -240,7 +272,7 @@ Note that the gradient is a function of \( \mathbf{x} =
|
||||
<li><a href="._Splines-bs019.html">20</a></li>
|
||||
<li><a href="._Splines-bs020.html">21</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs012.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -233,7 +265,7 @@ randomness. One such method is that of Stochastic Gradient Descent
|
||||
<li><a href="._Splines-bs020.html">21</a></li>
|
||||
<li><a href="._Splines-bs021.html">22</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs013.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -234,7 +266,7 @@ regular polygons (triangles, rectangles, pentagons, etc...).
|
||||
<li><a href="._Splines-bs021.html">22</a></li>
|
||||
<li><a href="._Splines-bs022.html">23</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs014.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -222,7 +254,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._Splines-bs022.html">23</a></li>
|
||||
<li><a href="._Splines-bs023.html">24</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs015.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -258,7 +290,7 @@ This condition is particularly useful since it gives us an procedure for determi
|
||||
<li><a href="._Splines-bs023.html">24</a></li>
|
||||
<li><a href="._Splines-bs024.html">25</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs016.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -245,7 +277,7 @@ This result means that if we know that the cost/loss function is convex and we a
|
||||
<li><a href="._Splines-bs024.html">25</a></li>
|
||||
<li><a href="._Splines-bs025.html">26</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs017.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -241,7 +273,7 @@ Using the definition of convexity, try to show that a function satisfying the pr
|
||||
<li><a href="._Splines-bs025.html">26</a></li>
|
||||
<li><a href="._Splines-bs026.html">27</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs018.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -248,7 +280,7 @@ When we have found the exact solution, \( \hat{r}=0 \).
|
||||
<li><a href="._Splines-bs026.html">27</a></li>
|
||||
<li><a href="._Splines-bs027.html">28</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs019.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -234,7 +266,7 @@ This quantity is always positive definite.
|
||||
<li><a href="._Splines-bs027.html">28</a></li>
|
||||
<li><a href="._Splines-bs028.html">29</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs020.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -237,7 +269,7 @@ instead.
|
||||
<li><a href="._Splines-bs028.html">29</a></li>
|
||||
<li><a href="._Splines-bs029.html">30</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs021.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -245,7 +277,7 @@ and
|
||||
<li><a href="._Splines-bs029.html">30</a></li>
|
||||
<li><a href="._Splines-bs030.html">31</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs022.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -241,7 +273,7 @@ $$
|
||||
<li><a href="._Splines-bs030.html">31</a></li>
|
||||
<li><a href="._Splines-bs031.html">32</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs023.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -253,7 +285,7 @@ $$
|
||||
<li><a href="._Splines-bs031.html">32</a></li>
|
||||
<li><a href="._Splines-bs032.html">33</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs024.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -191,7 +223,43 @@ MathJax.Hub.Config({
|
||||
<a name="part0024"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec23" class="anchor">The Steepest descent algorithm </h2>
|
||||
<h2 id="___sec23" class="anchor">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<p>
|
||||
|
||||
<!-- code=c++ (!bc cppcod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic"><cmath></span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic"><iostream></span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic"><fstream></span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic"><iomanip></span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic">"vectormatrixclass.h"</span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #008000; font-weight: bold">using</span> <span style="color: #008000; font-weight: bold">namespace</span> std;
|
||||
<span style="color: #408080; font-style: italic">// Main function begins here</span>
|
||||
<span style="color: #B00040">int</span> <span style="color: #0000FF">main</span>(<span style="color: #B00040">int</span> argc, <span style="color: #B00040">char</span> <span style="color: #666666">*</span> argv[]){
|
||||
<span style="color: #B00040">int</span> dim <span style="color: #666666">=</span> <span style="color: #666666">2</span>;
|
||||
Vector x(dim),xsd(dim), b(dim),x0(dim);
|
||||
Matrix A(dim,dim);
|
||||
|
||||
<span style="color: #408080; font-style: italic">// Set our initial guess</span>
|
||||
x0(<span style="color: #666666">0</span>) <span style="color: #666666">=</span> x0(<span style="color: #666666">1</span>) <span style="color: #666666">=</span> <span style="color: #666666">0</span>;
|
||||
<span style="color: #408080; font-style: italic">// Set the matrix</span>
|
||||
A(<span style="color: #666666">0</span>,<span style="color: #666666">0</span>) <span style="color: #666666">=</span> <span style="color: #666666">3</span>; A(<span style="color: #666666">1</span>,<span style="color: #666666">0</span>) <span style="color: #666666">=</span> <span style="color: #666666">2</span>; A(<span style="color: #666666">0</span>,<span style="color: #666666">1</span>) <span style="color: #666666">=</span> <span style="color: #666666">2</span>; A(<span style="color: #666666">1</span>,<span style="color: #666666">1</span>) <span style="color: #666666">=</span> <span style="color: #666666">6</span>;
|
||||
b(<span style="color: #666666">0</span>) <span style="color: #666666">=</span> <span style="color: #666666">2</span>; b(<span style="color: #666666">1</span>) <span style="color: #666666">=</span> <span style="color: #666666">-8</span>;
|
||||
cout <span style="color: #666666"><<</span> <span style="color: #BA2121">"The Matrix A that we are using: "</span> <span style="color: #666666"><<</span> endl;
|
||||
A.Print();
|
||||
cout <span style="color: #666666"><<</span> endl;
|
||||
xsd <span style="color: #666666">=</span> SteepestDescent(A,b,x0);
|
||||
cout <span style="color: #666666"><<</span> <span style="color: #BA2121">"The approximate solution using Steepest Descent is: "</span> <span style="color: #666666"><<</span> endl;
|
||||
xsd.Print();
|
||||
cout <span style="color: #666666"><<</span> endl;
|
||||
}
|
||||
</pre></div>
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -219,7 +287,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._Splines-bs032.html">33</a></li>
|
||||
<li><a href="._Splines-bs033.html">34</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs025.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -191,37 +223,33 @@ MathJax.Hub.Config({
|
||||
<a name="part0025"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec24" class="anchor">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
|
||||
<h2 id="___sec24" class="anchor">The routine for the steepest descent method </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<p>
|
||||
|
||||
<!-- code=c++ (!bc cppcod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic"><cmath></span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic"><iostream></span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic"><fstream></span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic"><iomanip></span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #BC7A00">#include</span> <span style="color: #408080; font-style: italic">"vectormatrixclass.h"</span><span style="color: #BC7A00"></span>
|
||||
<span style="color: #008000; font-weight: bold">using</span> <span style="color: #008000; font-weight: bold">namespace</span> std;
|
||||
<span style="color: #408080; font-style: italic">// Main function begins here</span>
|
||||
<span style="color: #B00040">int</span> <span style="color: #0000FF">main</span>(<span style="color: #B00040">int</span> argc, <span style="color: #B00040">char</span> <span style="color: #666666">*</span> argv[]){
|
||||
<span style="color: #B00040">int</span> dim <span style="color: #666666">=</span> <span style="color: #666666">2</span>;
|
||||
Vector x(dim),xsd(dim), b(dim),x0(dim);
|
||||
Matrix A(dim,dim);
|
||||
|
||||
<span style="color: #408080; font-style: italic">// Set our initial guess</span>
|
||||
x0(<span style="color: #666666">0</span>) <span style="color: #666666">=</span> x0(<span style="color: #666666">1</span>) <span style="color: #666666">=</span> <span style="color: #666666">0</span>;
|
||||
<span style="color: #408080; font-style: italic">// Set the matrix</span>
|
||||
A(<span style="color: #666666">0</span>,<span style="color: #666666">0</span>) <span style="color: #666666">=</span> <span style="color: #666666">3</span>; A(<span style="color: #666666">1</span>,<span style="color: #666666">0</span>) <span style="color: #666666">=</span> <span style="color: #666666">2</span>; A(<span style="color: #666666">0</span>,<span style="color: #666666">1</span>) <span style="color: #666666">=</span> <span style="color: #666666">2</span>; A(<span style="color: #666666">1</span>,<span style="color: #666666">1</span>) <span style="color: #666666">=</span> <span style="color: #666666">6</span>;
|
||||
b(<span style="color: #666666">0</span>) <span style="color: #666666">=</span> <span style="color: #666666">2</span>; b(<span style="color: #666666">1</span>) <span style="color: #666666">=</span> <span style="color: #666666">-8</span>;
|
||||
cout <span style="color: #666666"><<</span> <span style="color: #BA2121">"The Matrix A that we are using: "</span> <span style="color: #666666"><<</span> endl;
|
||||
A.Print();
|
||||
cout <span style="color: #666666"><<</span> endl;
|
||||
xsd <span style="color: #666666">=</span> SteepestDescent(A,b,x0);
|
||||
cout <span style="color: #666666"><<</span> <span style="color: #BA2121">"The approximate solution using Steepest Descent is: "</span> <span style="color: #666666"><<</span> endl;
|
||||
xsd.Print();
|
||||
cout <span style="color: #666666"><<</span> endl;
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>Vector <span style="color: #0000FF">SteepestDescent</span>(Matrix A, Vector b, Vector x0){
|
||||
<span style="color: #B00040">int</span> IterMax, i;
|
||||
<span style="color: #B00040">int</span> dim <span style="color: #666666">=</span> x0.Dimension();
|
||||
<span style="color: #008000; font-weight: bold">const</span> <span style="color: #B00040">double</span> tolerance <span style="color: #666666">=</span> <span style="color: #666666">1.0e-14</span>;
|
||||
Vector x(dim),f(dim),z(dim);
|
||||
<span style="color: #B00040">double</span> c,alpha,d;
|
||||
IterMax <span style="color: #666666">=</span> <span style="color: #666666">30</span>;
|
||||
x <span style="color: #666666">=</span> x0;
|
||||
f <span style="color: #666666">=</span> A<span style="color: #666666">*</span>x<span style="color: #666666">-</span>b;
|
||||
i <span style="color: #666666">=</span> <span style="color: #666666">0</span>;
|
||||
<span style="color: #008000; font-weight: bold">while</span> (i <span style="color: #666666"><=</span> IterMax){
|
||||
z <span style="color: #666666">=</span> A<span style="color: #666666">*</span>f;
|
||||
c <span style="color: #666666">=</span> dot(f,f);
|
||||
alpha <span style="color: #666666">=</span> c<span style="color: #666666">/</span>dot(f,z);
|
||||
x <span style="color: #666666">=</span> x <span style="color: #666666">-</span> alpha<span style="color: #666666">*</span>f;
|
||||
f <span style="color: #666666">=</span> A<span style="color: #666666">*</span>x<span style="color: #666666">-</span>b;
|
||||
<span style="color: #008000; font-weight: bold">if</span>(sqrt(dot(f,f)) <span style="color: #666666"><</span> tolerance) <span style="color: #008000; font-weight: bold">break</span>;
|
||||
i<span style="color: #666666">++</span>;
|
||||
}
|
||||
<span style="color: #008000; font-weight: bold">return</span> x;
|
||||
}
|
||||
</pre></div>
|
||||
<p>
|
||||
@@ -255,7 +283,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._Splines-bs033.html">34</a></li>
|
||||
<li><a href="._Splines-bs034.html">35</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs026.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -191,40 +223,72 @@ MathJax.Hub.Config({
|
||||
<a name="part0026"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec25" class="anchor">The routine for the steepest descent method </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<h2 id="___sec25" class="anchor">Steepest descent example </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=c++ (!bc cppcod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>Vector <span style="color: #0000FF">SteepestDescent</span>(Matrix A, Vector b, Vector x0){
|
||||
<span style="color: #B00040">int</span> IterMax, i;
|
||||
<span style="color: #B00040">int</span> dim <span style="color: #666666">=</span> x0.Dimension();
|
||||
<span style="color: #008000; font-weight: bold">const</span> <span style="color: #B00040">double</span> tolerance <span style="color: #666666">=</span> <span style="color: #666666">1.0e-14</span>;
|
||||
Vector x(dim),f(dim),z(dim);
|
||||
<span style="color: #B00040">double</span> c,alpha,d;
|
||||
IterMax <span style="color: #666666">=</span> <span style="color: #666666">30</span>;
|
||||
x <span style="color: #666666">=</span> x0;
|
||||
f <span style="color: #666666">=</span> A<span style="color: #666666">*</span>x<span style="color: #666666">-</span>b;
|
||||
i <span style="color: #666666">=</span> <span style="color: #666666">0</span>;
|
||||
<span style="color: #008000; font-weight: bold">while</span> (i <span style="color: #666666"><=</span> IterMax){
|
||||
z <span style="color: #666666">=</span> A<span style="color: #666666">*</span>f;
|
||||
c <span style="color: #666666">=</span> dot(f,f);
|
||||
alpha <span style="color: #666666">=</span> c<span style="color: #666666">/</span>dot(f,z);
|
||||
x <span style="color: #666666">=</span> x <span style="color: #666666">-</span> alpha<span style="color: #666666">*</span>f;
|
||||
f <span style="color: #666666">=</span> A<span style="color: #666666">*</span>x<span style="color: #666666">-</span>b;
|
||||
<span style="color: #008000; font-weight: bold">if</span>(sqrt(dot(f,f)) <span style="color: #666666"><</span> tolerance) <span style="color: #008000; font-weight: bold">break</span>;
|
||||
i<span style="color: #666666">++</span>;
|
||||
}
|
||||
<span style="color: #008000; font-weight: bold">return</span> x;
|
||||
}
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy.linalg</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">la</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scipy.optimize</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sopt</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pt</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> axes3d
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">0.5*</span>x[<span style="color: #666666">0</span>]<span style="color: #666666">**2</span> <span style="color: #666666">+</span> <span style="color: #666666">2.5*</span>x[<span style="color: #666666">1</span>]<span style="color: #666666">**2</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">df</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>array([x[<span style="color: #666666">0</span>], <span style="color: #666666">5*</span>x[<span style="color: #666666">1</span>]])
|
||||
|
||||
fig <span style="color: #666666">=</span> pt<span style="color: #666666">.</span>figure()
|
||||
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>gca(projection<span style="color: #666666">=</span><span style="color: #BA2121">"3d"</span>)
|
||||
|
||||
xmesh, ymesh <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mgrid[<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>,<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>]
|
||||
fmesh <span style="color: #666666">=</span> f(np<span style="color: #666666">.</span>array([xmesh, ymesh]))
|
||||
ax<span style="color: #666666">.</span>plot_surface(xmesh, ymesh, fmesh)
|
||||
</pre></div>
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
And then as countor plot
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">"equal"</span>)
|
||||
pt<span style="color: #666666">.</span>contour(xmesh, ymesh, fmesh)
|
||||
guesses <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">2./5</span>])]
|
||||
</pre></div>
|
||||
<p>
|
||||
Find guesses
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> guesses[<span style="color: #666666">-1</span>]
|
||||
s <span style="color: #666666">=</span> <span style="color: #666666">-</span>df(x)
|
||||
</pre></div>
|
||||
<p>
|
||||
Run it!
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1d</span>(alpha):
|
||||
<span style="color: #008000; font-weight: bold">return</span> f(x <span style="color: #666666">+</span> alpha<span style="color: #666666">*</span>s)
|
||||
|
||||
alpha_opt <span style="color: #666666">=</span> sopt<span style="color: #666666">.</span>golden(f1d)
|
||||
next_guess <span style="color: #666666">=</span> x <span style="color: #666666">+</span> alpha_opt <span style="color: #666666">*</span> s
|
||||
guesses<span style="color: #666666">.</span>append(next_guess)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(next_guess)
|
||||
</pre></div>
|
||||
<p>
|
||||
What happened?
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">"equal"</span>)
|
||||
pt<span style="color: #666666">.</span>contour(xmesh, ymesh, fmesh, <span style="color: #666666">50</span>)
|
||||
it_array <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array(guesses)
|
||||
pt<span style="color: #666666">.</span>plot(it_array<span style="color: #666666">.</span>T[<span style="color: #666666">0</span>], it_array<span style="color: #666666">.</span>T[<span style="color: #666666">1</span>], <span style="color: #BA2121">"x-"</span>)
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -251,7 +315,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._Splines-bs034.html">35</a></li>
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs027.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -247,7 +279,7 @@ $$
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs028.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -239,7 +271,7 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs029.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -229,7 +261,7 @@ where \( X \) is the design matrix defined above.
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs030.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -228,7 +260,7 @@ This result implies that \( C(\beta) \) is a convex function since the matrix \(
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs031.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -255,7 +287,7 @@ beta_NE <span style="color: #666666">=</span> np<span style="color: #666666">.</
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs032.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -262,6 +294,8 @@ plt<span style="color: #666666">.</span>show()
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs033.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -236,6 +268,9 @@ sgdreg<span style="color: #666666">.</span>fit(x,y<span style="color: #666666">.
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs034.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -268,6 +300,10 @@ beta_ridge <span style="color: #666666">=</span> np<span style="color: #666666">
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs035.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -191,21 +223,58 @@ MathJax.Hub.Config({
|
||||
<a name="part0035"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec34" class="anchor">Stochastic Gradient Descent </h2>
|
||||
|
||||
<p>
|
||||
Stochastic gradient descent (SGD) and variants thereof address some of
|
||||
the shortcomings of the Gradient descent method discussed above.
|
||||
|
||||
<p>
|
||||
The underlying idea of SGD comes from the observation that the cost
|
||||
function, which we want to minimize, can almost always be written as a
|
||||
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
|
||||
<h2 id="___sec34" class="anchor">Automatic differentiation </h2>
|
||||
Python has tools for so-called <b>automatic differentiation</b>.
|
||||
Consider the following example
|
||||
$$
|
||||
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
|
||||
\mathbf{\beta}).
|
||||
f(x) = \sin\left(2\pi x + x^2\right)
|
||||
$$
|
||||
|
||||
which has the following derivative
|
||||
$$
|
||||
f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right)
|
||||
$$
|
||||
|
||||
Using <b>autograd</b> we have
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># To do elementwise differentiation:</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> elementwise_grad <span style="color: #008000; font-weight: bold">as</span> egrad
|
||||
|
||||
<span style="color: #408080; font-style: italic"># To plot:</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>sin(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f_grad_analytic</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>cos(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>)<span style="color: #666666">*</span>(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi <span style="color: #666666">+</span> <span style="color: #666666">2*</span>x)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Do the comparison:</span>
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">0</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1000</span>)
|
||||
|
||||
f_grad <span style="color: #666666">=</span> egrad(f)
|
||||
|
||||
computed <span style="color: #666666">=</span> f_grad(x)
|
||||
analytic <span style="color: #666666">=</span> f_grad_analytic(x)
|
||||
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">'Derivative computed from Autograd compared with the analytical derivative'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x,computed,label<span style="color: #666666">=</span><span style="color: #BA2121">'autograd'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x,analytic,label<span style="color: #666666">=</span><span style="color: #BA2121">'analytic'</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">'x'</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">'y'</span>)
|
||||
plt<span style="color: #666666">.</span>legend()
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The max absolute difference is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(np<span style="color: #666666">.</span>max(np<span style="color: #666666">.</span>abs(computed <span style="color: #666666">-</span> analytic))))
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -228,6 +297,11 @@ $$
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs036.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -189,25 +221,38 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0036"></a>
|
||||
<!-- !split -->
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec35" class="anchor">Computation of gradients </h2>
|
||||
<h2 id="___sec35" class="anchor">Using autograd </h2>
|
||||
|
||||
<p>
|
||||
This in turn means that the gradient can be
|
||||
computed as a sum over \( i \)-gradients
|
||||
$$
|
||||
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
|
||||
\mathbf{\beta}).
|
||||
$$
|
||||
Here we
|
||||
experiment with what kind of functions Autograd is capable
|
||||
of finding the gradient of. The following Python functions are just
|
||||
meant to illustrate what Autograd can do, but please feel free to
|
||||
experiment with other, possibly more complicated, functions as well.
|
||||
|
||||
<p>
|
||||
Stochasticity/randomness is introduced by only taking the
|
||||
gradient on a subset of the data called minibatches. If there are \( n \)
|
||||
data points and the size of each minibatch is \( M \), there will be \( n/M \)
|
||||
minibatches. We denote these minibatches by \( B_k \) where
|
||||
\( k=1,\cdots,n/M \).
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">**3</span> <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
|
||||
f1_grad <span style="color: #666666">=</span> grad(f1)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Remember to send in float as argument to the computed gradient from Autograd!</span>
|
||||
a <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># See the evaluated gradient at a using autograd:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> using autograd is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(a,f1_grad(a)))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Compare with the analytical derivative, that is f1'(x) = 3*x**2 </span>
|
||||
grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">3*</span>a<span style="color: #666666">**2</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> by finding the analytic expression is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(a,grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -229,6 +274,12 @@ minibatches. We denote these minibatches by \( B_k \) where
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs037.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -0,0 +1,324 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0037"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec36" class="anchor">Autograd with more complicated functions </h2>
|
||||
|
||||
<p>
|
||||
To differentiate with respect to two (or more) arguments of a Python
|
||||
function, Autograd need to know at which variable the function if
|
||||
being differentiated with respect to.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f2</span>(x1,x2):
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">3*</span>x1<span style="color: #666666">**3</span> <span style="color: #666666">+</span> x2<span style="color: #666666">*</span>(x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># By sending the argument 0, Autograd will compute the derivative w.r.t the first variable, in this case x1</span>
|
||||
f2_grad_x1 <span style="color: #666666">=</span> grad(f2,<span style="color: #666666">0</span>)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># ... and differentiate w.r.t x2 by sending 1 as an additional arugment to grad</span>
|
||||
f2_grad_x2 <span style="color: #666666">=</span> grad(f2,<span style="color: #666666">1</span>)
|
||||
|
||||
x1 <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
|
||||
x2 <span style="color: #666666">=</span> <span style="color: #666666">3.0</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Evaluating at x1 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">, x2 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x1,x2))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"-"</span><span style="color: #666666">*30</span>)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Compare with the analytical derivatives:</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Derivative of f2 w.r.t x1 is: 9*x1**2 + x2:</span>
|
||||
f2_grad_x1_analytical <span style="color: #666666">=</span> <span style="color: #666666">9*</span>x1<span style="color: #666666">**2</span> <span style="color: #666666">+</span> x2
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Derivative of f2 w.r.t x2 is: x1 - 5:</span>
|
||||
f2_grad_x2_analytical <span style="color: #666666">=</span> x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># See the evaluated derivations:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>()
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs036.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs029.html">30</a></li>
|
||||
<li><a href="._Splines-bs030.html">31</a></li>
|
||||
<li><a href="._Splines-bs031.html">32</a></li>
|
||||
<li><a href="._Splines-bs032.html">33</a></li>
|
||||
<li><a href="._Splines-bs033.html">34</a></li>
|
||||
<li><a href="._Splines-bs034.html">35</a></li>
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li class="active"><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs038.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0038"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec37" class="anchor">More complicated functions using the elements of their arguments directly </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f3</span>(x): <span style="color: #408080; font-style: italic"># Assumes x is an array of length 5 or higher</span>
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">2*</span>x[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> <span style="color: #666666">3*</span>x[<span style="color: #666666">1</span>] <span style="color: #666666">+</span> <span style="color: #666666">5*</span>x[<span style="color: #666666">2</span>] <span style="color: #666666">+</span> <span style="color: #666666">7*</span>x[<span style="color: #666666">3</span>] <span style="color: #666666">+</span> <span style="color: #666666">11*</span>x[<span style="color: #666666">4</span>]<span style="color: #666666">**2</span>
|
||||
|
||||
f3_grad <span style="color: #666666">=</span> grad(f3)
|
||||
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">0</span>,<span style="color: #666666">4</span>,<span style="color: #666666">5</span>)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the computed gradient:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed gradient of f3 is: "</span>, f3_grad(x))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
|
||||
f3_grad_analytical <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">5</span>, <span style="color: #666666">7</span>, <span style="color: #666666">22*</span>x[<span style="color: #666666">4</span>]])
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical gradient of f3 is: "</span>, f3_grad_analytical)
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that in this case, when sending an array as input argument, the
|
||||
output from Autograd is another array. This is the true gradient of
|
||||
the function, as opposed to the function in the previous example. By
|
||||
using arrays to represent the variables, the output from Autograd
|
||||
might be easier to work with, as the output is closer to what one
|
||||
could expect form a gradient-evaluting function.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs037.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs030.html">31</a></li>
|
||||
<li><a href="._Splines-bs031.html">32</a></li>
|
||||
<li><a href="._Splines-bs032.html">33</a></li>
|
||||
<li><a href="._Splines-bs033.html">34</a></li>
|
||||
<li><a href="._Splines-bs034.html">35</a></li>
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li class="active"><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs039.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0039"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec38" class="anchor">Functions using mathematical functions from Numpy </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f4</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1+</span>x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>sin(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)
|
||||
|
||||
f4_grad <span style="color: #666666">=</span> grad(f4)
|
||||
|
||||
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f4_grad(x)))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
|
||||
f4_grad_analytical <span style="color: #666666">=</span> x<span style="color: #666666">/</span>np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1</span> <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>cos(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)<span style="color: #666666">*2*</span>np<span style="color: #666666">.</span>pi
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical gradient of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f4_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs038.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs031.html">32</a></li>
|
||||
<li><a href="._Splines-bs032.html">33</a></li>
|
||||
<li><a href="._Splines-bs033.html">34</a></li>
|
||||
<li><a href="._Splines-bs034.html">35</a></li>
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li class="active"><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs040.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0040"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec39" class="anchor">More autograd </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f5</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">if</span> x <span style="color: #666666">>=</span> <span style="color: #666666">0</span>:
|
||||
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">**2</span>
|
||||
<span style="color: #008000; font-weight: bold">else</span>:
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">-3*</span>x <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
|
||||
f5_grad <span style="color: #666666">=</span> grad(f5)
|
||||
|
||||
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f5 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f5_grad(x)))
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs039.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs032.html">33</a></li>
|
||||
<li><a href="._Splines-bs033.html">34</a></li>
|
||||
<li><a href="._Splines-bs034.html">35</a></li>
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li class="active"><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs041.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,320 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0041"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec40" class="anchor">And with loops </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f6_for</span>(x):
|
||||
val <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10</span>):
|
||||
val <span style="color: #666666">=</span> val <span style="color: #666666">+</span> x<span style="color: #666666">**</span>i
|
||||
<span style="color: #008000; font-weight: bold">return</span> val
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f6_while</span>(x):
|
||||
val <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
i <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">while</span> i <span style="color: #666666"><</span> <span style="color: #666666">10</span>:
|
||||
val <span style="color: #666666">=</span> val <span style="color: #666666">+</span> x<span style="color: #666666">**</span>i
|
||||
i <span style="color: #666666">=</span> i <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
<span style="color: #008000; font-weight: bold">return</span> val
|
||||
|
||||
f6_for_grad <span style="color: #666666">=</span> grad(f6_for)
|
||||
f6_while_grad <span style="color: #666666">=</span> grad(f6_while)
|
||||
|
||||
x <span style="color: #666666">=</span> <span style="color: #666666">0.5</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the computed derivaties of f6_for and f6_while</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f6_for at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f6_for_grad(x)))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f6_while at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f6_while_grad(x)))
|
||||
</pre></div>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #408080; font-style: italic"># Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9</span>
|
||||
<span style="color: #408080; font-style: italic"># The analytical derivative is: sum(i*x**(i-1)) </span>
|
||||
f6_grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10</span>):
|
||||
f6_grad_analytical <span style="color: #666666">+=</span> i<span style="color: #666666">*</span>x<span style="color: #666666">**</span>(i<span style="color: #666666">-1</span>)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical derivative of f6 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f6_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs040.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs033.html">34</a></li>
|
||||
<li><a href="._Splines-bs034.html">35</a></li>
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li class="active"><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs042.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0042"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec41" class="anchor">Using recursion </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f7</span>(n): <span style="color: #408080; font-style: italic"># Assume that n is an integer</span>
|
||||
<span style="color: #008000; font-weight: bold">if</span> n <span style="color: #666666">==</span> <span style="color: #666666">1</span> <span style="color: #AA22FF; font-weight: bold">or</span> n <span style="color: #666666">==</span> <span style="color: #666666">0</span>:
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1</span>
|
||||
<span style="color: #008000; font-weight: bold">else</span>:
|
||||
<span style="color: #008000; font-weight: bold">return</span> n<span style="color: #666666">*</span>f7(n<span style="color: #666666">-1</span>)
|
||||
|
||||
f7_grad <span style="color: #666666">=</span> grad(f7)
|
||||
|
||||
n <span style="color: #666666">=</span> <span style="color: #666666">2.0</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(n,f7_grad(n)))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># The function f7 is an implementation of the factorial of n.</span>
|
||||
<span style="color: #408080; font-style: italic"># By using the product rule, one can find that the derivative is:</span>
|
||||
|
||||
f7_grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">int</span>(n)<span style="color: #666666">-1</span>):
|
||||
tmp <span style="color: #666666">=</span> <span style="color: #666666">1</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">int</span>(n)<span style="color: #666666">-1</span>):
|
||||
<span style="color: #008000; font-weight: bold">if</span> k <span style="color: #666666">!=</span> i:
|
||||
tmp <span style="color: #666666">*=</span> (n <span style="color: #666666">-</span> k)
|
||||
f7_grad_analytical <span style="color: #666666">+=</span> tmp
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(n,f7_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs041.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs034.html">35</a></li>
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li class="active"><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs043.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,298 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0043"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec42" class="anchor">Unsupported functions </h2>
|
||||
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
|
||||
|
||||
<p>
|
||||
Assigning a value to the variable being differentiated with respect to
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f8</span>(x): <span style="color: #408080; font-style: italic"># Assume x is an array</span>
|
||||
x[<span style="color: #666666">2</span>] <span style="color: #666666">=</span> <span style="color: #666666">3</span>
|
||||
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">*2</span>
|
||||
|
||||
f8_grad <span style="color: #666666">=</span> grad(f8)
|
||||
|
||||
x <span style="color: #666666">=</span> <span style="color: #666666">8.4</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The derivative of f8 is:"</span>,f8_grad(x))
|
||||
</pre></div>
|
||||
<p>
|
||||
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs042.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs035.html">36</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li class="active"><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs044.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,313 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0044"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec43" class="anchor">The syntax a.dot(b) when finding the dot product </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f9</span>(a): <span style="color: #408080; font-style: italic"># Assume a is an array with 2 elements</span>
|
||||
b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">2.0</span>])
|
||||
<span style="color: #008000; font-weight: bold">return</span> a<span style="color: #666666">.</span>dot(b)
|
||||
|
||||
f9_grad <span style="color: #666666">=</span> grad(f9)
|
||||
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">0.0</span>])
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The derivative of f9 is:"</span>,f9_grad(x))
|
||||
</pre></div>
|
||||
<p>
|
||||
Here we are told that the 'dot' function does not belong to Autograd's
|
||||
version of a Numpy array. To overcome this, an alternative syntax
|
||||
which also computed the dot product can be used:
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f9_alternative</span>(x): <span style="color: #408080; font-style: italic"># Assume a is an array with 2 elements</span>
|
||||
b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">2.0</span>])
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>dot(x,b) <span style="color: #408080; font-style: italic"># The same as x_1*b_1 + x_2*b_2</span>
|
||||
|
||||
f9_alternative_grad <span style="color: #666666">=</span> grad(f9_alternative)
|
||||
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">3.0</span>,<span style="color: #666666">0.0</span>])
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The gradient of f9 is:"</span>,f9_alternative_grad(x))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
|
||||
<span style="color: #408080; font-style: italic"># w.r.t x is (b_1, b_2).</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs043.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs036.html">37</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li class="active"><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs045.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0045"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec44" class="anchor">Recommended to avoid </h2>
|
||||
The documentation recommends to avoid inplace operations such as
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>a <span style="color: #666666">+=</span> b
|
||||
a <span style="color: #666666">-=</span> b
|
||||
a<span style="color: #666666">*=</span> b
|
||||
a <span style="color: #666666">/=</span>b
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs044.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs037.html">38</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li class="active"><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs046.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,287 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0046"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec45" class="anchor">Stochastic Gradient Descent </h2>
|
||||
|
||||
<p>
|
||||
Stochastic gradient descent (SGD) and variants thereof address some of
|
||||
the shortcomings of the Gradient descent method discussed above.
|
||||
|
||||
<p>
|
||||
The underlying idea of SGD comes from the observation that the cost
|
||||
function, which we want to minimize, can almost always be written as a
|
||||
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
|
||||
$$
|
||||
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
|
||||
\mathbf{\beta}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs045.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs038.html">39</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li class="active"><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs047.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0047"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec46" class="anchor">Computation of gradients </h2>
|
||||
|
||||
<p>
|
||||
This in turn means that the gradient can be
|
||||
computed as a sum over \( i \)-gradients
|
||||
$$
|
||||
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
|
||||
\mathbf{\beta}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
Stochasticity/randomness is introduced by only taking the
|
||||
gradient on a subset of the data called minibatches. If there are \( n \)
|
||||
data points and the size of each minibatch is \( M \), there will be \( n/M \)
|
||||
minibatches. We denote these minibatches by \( B_k \) where
|
||||
\( k=1,\cdots,n/M \).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs046.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs039.html">40</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li class="active"><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs048.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,291 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0048"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec47" class="anchor">SGD example </h2>
|
||||
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
|
||||
and we choose to have \( M=5 \) minibathces,
|
||||
then each minibatch contains two data points. In particular we have
|
||||
\( B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
|
||||
(\mathbf{x}_9,\mathbf{x}_{10}) \). Note that if you choose \( M=1 \) you
|
||||
have only a single batch with all data points and on the other extreme,
|
||||
you may choose \( M=n \) resulting in a minibatch for each datapoint, i.e
|
||||
\( B_k = \mathbf{x}_k \).
|
||||
|
||||
<p>
|
||||
The idea is now to approximate the gradient by replacing the sum over
|
||||
all data points with a sum over the data points in one the minibatches
|
||||
picked at random in each gradient descent step
|
||||
$$
|
||||
\nabla_{\beta}
|
||||
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
|
||||
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
|
||||
c_i(\mathbf{x}_i, \mathbf{\beta}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs047.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs040.html">41</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li class="active"><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs049.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0049"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec48" class="anchor">The gradient step </h2>
|
||||
|
||||
<p>
|
||||
Thus a gradient descent step now looks like
|
||||
$$
|
||||
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
|
||||
\mathbf{\beta})
|
||||
$$
|
||||
|
||||
<p>
|
||||
where \( k \) is picked at random with equal
|
||||
probability from \( [1,n/M] \). An iteration over the number of
|
||||
minibathces (n/M) is commonly referred to as an epoch. Thus it is
|
||||
typical to choose a number of epochs and for each epoch iterate over
|
||||
the number of minibatches, as exemplified in the code below.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs048.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li class="active"><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs050.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0050"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec49" class="anchor">Simple example code </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
|
||||
n <span style="color: #666666">=</span> <span style="color: #666666">100</span> <span style="color: #408080; font-style: italic">#100 datapoints </span>
|
||||
M <span style="color: #666666">=</span> <span style="color: #666666">5</span> <span style="color: #408080; font-style: italic">#size of each minibatch</span>
|
||||
m <span style="color: #666666">=</span> <span style="color: #008000">int</span>(n<span style="color: #666666">/</span>M) <span style="color: #408080; font-style: italic">#number of minibatches</span>
|
||||
n_epochs <span style="color: #666666">=</span> <span style="color: #666666">10</span> <span style="color: #408080; font-style: italic">#number of epochs</span>
|
||||
|
||||
j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> epoch <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">1</span>,n_epochs<span style="color: #666666">+1</span>):
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(m):
|
||||
k <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randint(m) <span style="color: #408080; font-style: italic">#Pick the k-th minibatch at random</span>
|
||||
<span style="color: #408080; font-style: italic">#Compute the gradient using the data in minibatch Bk</span>
|
||||
<span style="color: #408080; font-style: italic">#Compute new suggestion for </span>
|
||||
j <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
Taking the gradient only on a subset of the data has two important
|
||||
benefits. First, it introduces randomness which decreases the chance
|
||||
that our opmization scheme gets stuck in a local minima. Second, if
|
||||
the size of the minibatches are small relative to the number of
|
||||
datapoints (\( M < n \)), the computation of the gradient is much
|
||||
cheaper since we sum over the datapoints in the \( k-th \) minibatch and not
|
||||
all \( n \) datapoints.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs049.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs042.html">43</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li class="active"><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs051.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0051"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec50" class="anchor">When do we stop? </h2>
|
||||
|
||||
<p>
|
||||
A natural question is when do we stop the search for a new minimum?
|
||||
One possibility is to compute the full gradient after a given number
|
||||
of epochs and check if the norm of the gradient is smaller than some
|
||||
threshold and stop if true. However, the condition that the gradient
|
||||
is zero is valid also for local minima, so this would only tell us
|
||||
that we are close to a local/global minimum. However, we could also
|
||||
evaluate the cost function at this point, store the result and
|
||||
continue the search. If the test kicks in at a later stage we can
|
||||
compare the values of the cost function and keep the \( \beta \) that
|
||||
gave the lowest value.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs050.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs043.html">44</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li class="active"><a href="._Splines-bs051.html">52</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs052.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,311 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods">
|
||||
|
||||
<title>Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Optimization, the central part of any Machine Learning '
|
||||
'algortithm',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Revisiting our Logistic Regression case', 2, None, '___sec1'),
|
||||
('The equations to solve', 2, None, '___sec2'),
|
||||
("Solving using Newton-Raphson's method", 2, None, '___sec3'),
|
||||
("Brief reminder on Newton-Raphson's method", 2, None, '___sec4'),
|
||||
('The equations', 2, None, '___sec5'),
|
||||
('Simple geometric interpretation', 2, None, '___sec6'),
|
||||
('Extending to more than one variable', 2, None, '___sec7'),
|
||||
('Steepest descent', 2, None, '___sec8'),
|
||||
('More on Steepest descent', 2, None, '___sec9'),
|
||||
('The ideal', 2, None, '___sec10'),
|
||||
('The sensitiveness of the gradient descent',
|
||||
2,
|
||||
None,
|
||||
'___sec11'),
|
||||
('Convex functions', 2, None, '___sec12'),
|
||||
('Convex function', 2, None, '___sec13'),
|
||||
('Conditions on convex functions', 2, None, '___sec14'),
|
||||
('More on convex functions', 2, None, '___sec15'),
|
||||
('Some simple problems', 2, None, '___sec16'),
|
||||
('Standard steepest descent', 2, None, '___sec17'),
|
||||
('Gradient method', 2, None, '___sec18'),
|
||||
('Steepest descent method', 2, None, '___sec19'),
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
('The Hessian matrix', 2, None, '___sec29'),
|
||||
('Simple program', 2, None, '___sec30'),
|
||||
('Gradient Descent Example', 2, None, '___sec31'),
|
||||
('And a corresponding example using _scikit-learn_',
|
||||
2,
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Splines-bs.html">Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs001.html#___sec0" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs002.html#___sec1" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs003.html#___sec2" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs004.html#___sec3" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs005.html#___sec4" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs006.html#___sec5" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs007.html#___sec6" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs008.html#___sec7" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs009.html#___sec8" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs010.html#___sec9" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs011.html#___sec10" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs012.html#___sec11" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs013.html#___sec12" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs014.html#___sec13" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs015.html#___sec14" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs016.html#___sec15" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs017.html#___sec16" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs018.html#___sec17" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs019.html#___sec18" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0052"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec51" class="anchor">Slightly different approach </h2>
|
||||
|
||||
<p>
|
||||
Another approach is to let the step length \( \gamma_j \) depend on the
|
||||
number of epochs in such a way that it becomes very small after a
|
||||
reasonable time such that we do not move at all.
|
||||
|
||||
<p>
|
||||
As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in <em>time</em> \( t \).
|
||||
|
||||
<p>
|
||||
In this way we can fix the number of epochs, compute \( \beta \) and
|
||||
evaluate the cost function at the end. Repeating the computation will
|
||||
give a different result since the scheme is random by design. Then we
|
||||
pick the final \( \beta \) that gives the lowest value of the cost
|
||||
function.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">step_length</span>(t,t0,t1):
|
||||
<span style="color: #008000; font-weight: bold">return</span> t0<span style="color: #666666">/</span>(t<span style="color: #666666">+</span>t1)
|
||||
|
||||
n <span style="color: #666666">=</span> <span style="color: #666666">100</span> <span style="color: #408080; font-style: italic">#100 datapoints </span>
|
||||
M <span style="color: #666666">=</span> <span style="color: #666666">5</span> <span style="color: #408080; font-style: italic">#size of each minibatch</span>
|
||||
m <span style="color: #666666">=</span> <span style="color: #008000">int</span>(n<span style="color: #666666">/</span>M) <span style="color: #408080; font-style: italic">#number of minibatches</span>
|
||||
n_epochs <span style="color: #666666">=</span> <span style="color: #666666">500</span> <span style="color: #408080; font-style: italic">#number of epochs</span>
|
||||
t0 <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
|
||||
t1 <span style="color: #666666">=</span> <span style="color: #666666">10</span>
|
||||
|
||||
gamma_j <span style="color: #666666">=</span> t0<span style="color: #666666">/</span>t1
|
||||
j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> epoch <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">1</span>,n_epochs<span style="color: #666666">+1</span>):
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(m):
|
||||
k <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randint(m) <span style="color: #408080; font-style: italic">#Pick the k-th minibatch at random</span>
|
||||
<span style="color: #408080; font-style: italic">#Compute the gradient using the data in minibatch Bk</span>
|
||||
<span style="color: #408080; font-style: italic">#Compute new suggestion for beta</span>
|
||||
t <span style="color: #666666">=</span> epoch<span style="color: #666666">*</span>m<span style="color: #666666">+</span>i
|
||||
gamma_j <span style="color: #666666">=</span> step_length(t,t0,t1)
|
||||
j <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"gamma_j after </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> epochs: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span> <span style="color: #666666">%</span> (n_epochs,gamma_j))
|
||||
</pre></div>
|
||||
<p>
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._Splines-bs051.html">«</a></li>
|
||||
<li><a href="._Splines-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs044.html">45</a></li>
|
||||
<li><a href="._Splines-bs045.html">46</a></li>
|
||||
<li><a href="._Splines-bs046.html">47</a></li>
|
||||
<li><a href="._Splines-bs047.html">48</a></li>
|
||||
<li><a href="._Splines-bs048.html">49</a></li>
|
||||
<li><a href="._Splines-bs049.html">50</a></li>
|
||||
<li><a href="._Splines-bs050.html">51</a></li>
|
||||
<li><a href="._Splines-bs051.html">52</a></li>
|
||||
<li class="active"><a href="._Splines-bs052.html">53</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -70,16 +70,16 @@ Automatically generated HTML file from DocOnce source
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -91,13 +91,34 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -158,9 +179,9 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Gradient descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The Steepest descent algorithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
@@ -169,13 +190,24 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs049.html#___sec48" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -234,7 +266,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._Splines-bs008.html">9</a></li>
|
||||
<li><a href="._Splines-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._Splines-bs041.html">42</a></li>
|
||||
<li><a href="._Splines-bs052.html">53</a></li>
|
||||
<li><a href="._Splines-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -818,12 +818,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec23">The Steepest descent algorithm </h2>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec24">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
|
||||
<h2 id="___sec23">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -861,7 +856,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec25">The routine for the steepest descent method </h2>
|
||||
<h2 id="___sec24">The routine for the steepest descent method </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -894,6 +889,76 @@ $$
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec25">Steepest descent example </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy.linalg</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">la</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">scipy.optimize</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sopt</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pt</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">mpl_toolkits.mplot3d</span> <span style="color: #8B008B; font-weight: bold">import</span> axes3d
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">0.5</span>*x[<span style="color: #B452CD">0</span>]**<span style="color: #B452CD">2</span> + <span style="color: #B452CD">2.5</span>*x[<span style="color: #B452CD">1</span>]**<span style="color: #B452CD">2</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">df</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.array([x[<span style="color: #B452CD">0</span>], <span style="color: #B452CD">5</span>*x[<span style="color: #B452CD">1</span>]])
|
||||
|
||||
fig = pt.figure()
|
||||
ax = fig.gca(projection=<span style="color: #CD5555">"3d"</span>)
|
||||
|
||||
xmesh, ymesh = np.mgrid[-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50j</span>,-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50j</span>]
|
||||
fmesh = f(np.array([xmesh, ymesh]))
|
||||
ax.plot_surface(xmesh, ymesh, fmesh)
|
||||
</pre></div>
|
||||
<p>
|
||||
And then as countor plot
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>pt.axis(<span style="color: #CD5555">"equal"</span>)
|
||||
pt.contour(xmesh, ymesh, fmesh)
|
||||
guesses = [np.array([<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2.</span>/<span style="color: #B452CD">5</span>])]
|
||||
</pre></div>
|
||||
<p>
|
||||
Find guesses
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>x = guesses[-<span style="color: #B452CD">1</span>]
|
||||
s = -df(x)
|
||||
</pre></div>
|
||||
<p>
|
||||
Run it!
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1d</span>(alpha):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> f(x + alpha*s)
|
||||
|
||||
alpha_opt = sopt.golden(f1d)
|
||||
next_guess = x + alpha_opt * s
|
||||
guesses.append(next_guess)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(next_guess)
|
||||
</pre></div>
|
||||
<p>
|
||||
What happened?
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>pt.axis(<span style="color: #CD5555">"equal"</span>)
|
||||
pt.contour(xmesh, ymesh, fmesh, <span style="color: #B452CD">50</span>)
|
||||
it_array = np.array(guesses)
|
||||
pt.plot(it_array.T[<span style="color: #B452CD">0</span>], it_array.T[<span style="color: #B452CD">1</span>], <span style="color: #CD5555">"x-"</span>)
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec26">Revisiting our first homework </h2>
|
||||
|
||||
@@ -1180,7 +1245,404 @@ beta_ridge = np.dot(Z,np.dot(X.T,y))
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec34">Stochastic Gradient Descent </h2>
|
||||
<h2 id="___sec34">Automatic differentiation </h2>
|
||||
Python has tools for so-called <b>automatic differentiation</b>.
|
||||
Consider the following example
|
||||
<p> <br>
|
||||
$$
|
||||
f(x) = \sin\left(2\pi x + x^2\right)
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
which has the following derivative
|
||||
<p> <br>
|
||||
$$
|
||||
f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right)
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
Using <b>autograd</b> we have
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
|
||||
<span style="color: #228B22"># To do elementwise differentiation:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> elementwise_grad <span style="color: #8B008B; font-weight: bold">as</span> egrad
|
||||
|
||||
<span style="color: #228B22"># To plot:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
||||
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.sin(<span style="color: #B452CD">2</span>*np.pi*x + x**<span style="color: #B452CD">2</span>)
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f_grad_analytic</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.cos(<span style="color: #B452CD">2</span>*np.pi*x + x**<span style="color: #B452CD">2</span>)*(<span style="color: #B452CD">2</span>*np.pi + <span style="color: #B452CD">2</span>*x)
|
||||
|
||||
<span style="color: #228B22"># Do the comparison:</span>
|
||||
x = np.linspace(<span style="color: #B452CD">0</span>,<span style="color: #B452CD">1</span>,<span style="color: #B452CD">1000</span>)
|
||||
|
||||
f_grad = egrad(f)
|
||||
|
||||
computed = f_grad(x)
|
||||
analytic = f_grad_analytic(x)
|
||||
|
||||
plt.title(<span style="color: #CD5555">'Derivative computed from Autograd compared with the analytical derivative'</span>)
|
||||
plt.plot(x,computed,label=<span style="color: #CD5555">'autograd'</span>)
|
||||
plt.plot(x,analytic,label=<span style="color: #CD5555">'analytic'</span>)
|
||||
|
||||
plt.xlabel(<span style="color: #CD5555">'x'</span>)
|
||||
plt.ylabel(<span style="color: #CD5555">'y'</span>)
|
||||
plt.legend()
|
||||
|
||||
plt.show()
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The max absolute difference is: %g"</span>%(np.max(np.abs(computed - analytic))))
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec35">Using autograd </h2>
|
||||
|
||||
<p>
|
||||
Here we
|
||||
experiment with what kind of functions Autograd is capable
|
||||
of finding the gradient of. The following Python functions are just
|
||||
meant to illustrate what Autograd can do, but please feel free to
|
||||
experiment with other, possibly more complicated, functions as well.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> x**<span style="color: #B452CD">3</span> + <span style="color: #B452CD">1</span>
|
||||
|
||||
f1_grad = grad(f1)
|
||||
|
||||
<span style="color: #228B22"># Remember to send in float as argument to the computed gradient from Autograd!</span>
|
||||
a = <span style="color: #B452CD">1.0</span>
|
||||
|
||||
<span style="color: #228B22"># See the evaluated gradient at a using autograd:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The gradient of f1 evaluated at a = %g using autograd is: %g"</span>%(a,f1_grad(a)))
|
||||
|
||||
<span style="color: #228B22"># Compare with the analytical derivative, that is f1'(x) = 3*x**2 </span>
|
||||
grad_analytical = <span style="color: #B452CD">3</span>*a**<span style="color: #B452CD">2</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g"</span>%(a,grad_analytical))
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec36">Autograd with more complicated functions </h2>
|
||||
|
||||
<p>
|
||||
To differentiate with respect to two (or more) arguments of a Python
|
||||
function, Autograd need to know at which variable the function if
|
||||
being differentiated with respect to.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f2</span>(x1,x2):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">3</span>*x1**<span style="color: #B452CD">3</span> + x2*(x1 - <span style="color: #B452CD">5</span>) + <span style="color: #B452CD">1</span>
|
||||
|
||||
<span style="color: #228B22"># By sending the argument 0, Autograd will compute the derivative w.r.t the first variable, in this case x1</span>
|
||||
f2_grad_x1 = grad(f2,<span style="color: #B452CD">0</span>)
|
||||
|
||||
<span style="color: #228B22"># ... and differentiate w.r.t x2 by sending 1 as an additional arugment to grad</span>
|
||||
f2_grad_x2 = grad(f2,<span style="color: #B452CD">1</span>)
|
||||
|
||||
x1 = <span style="color: #B452CD">1.0</span>
|
||||
x2 = <span style="color: #B452CD">3.0</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"Evaluating at x1 = %g, x2 = %g"</span>%(x1,x2))
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"-"</span>*<span style="color: #B452CD">30</span>)
|
||||
|
||||
<span style="color: #228B22"># Compare with the analytical derivatives:</span>
|
||||
|
||||
<span style="color: #228B22"># Derivative of f2 w.r.t x1 is: 9*x1**2 + x2:</span>
|
||||
f2_grad_x1_analytical = <span style="color: #B452CD">9</span>*x1**<span style="color: #B452CD">2</span> + x2
|
||||
|
||||
<span style="color: #228B22"># Derivative of f2 w.r.t x2 is: x1 - 5:</span>
|
||||
f2_grad_x2_analytical = x1 - <span style="color: #B452CD">5</span>
|
||||
|
||||
<span style="color: #228B22"># See the evaluated derivations:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The derivative of f2 w.r.t x1: %g"</span>%( f2_grad_x1(x1,x2) ))
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical derivative of f2 w.r.t x1: %g"</span>%( f2_grad_x1(x1,x2) ))
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>()
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The derivative of f2 w.r.t x2: %g"</span>%( f2_grad_x2(x1,x2) ))
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical derivative of f2 w.r.t x2: %g"</span>%( f2_grad_x2(x1,x2) ))
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec37">More complicated functions using the elements of their arguments directly </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f3</span>(x): <span style="color: #228B22"># Assumes x is an array of length 5 or higher</span>
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">2</span>*x[<span style="color: #B452CD">0</span>] + <span style="color: #B452CD">3</span>*x[<span style="color: #B452CD">1</span>] + <span style="color: #B452CD">5</span>*x[<span style="color: #B452CD">2</span>] + <span style="color: #B452CD">7</span>*x[<span style="color: #B452CD">3</span>] + <span style="color: #B452CD">11</span>*x[<span style="color: #B452CD">4</span>]**<span style="color: #B452CD">2</span>
|
||||
|
||||
f3_grad = grad(f3)
|
||||
|
||||
x = np.linspace(<span style="color: #B452CD">0</span>,<span style="color: #B452CD">4</span>,<span style="color: #B452CD">5</span>)
|
||||
|
||||
<span style="color: #228B22"># Print the computed gradient:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed gradient of f3 is: "</span>, f3_grad(x))
|
||||
|
||||
<span style="color: #228B22"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
|
||||
f3_grad_analytical = np.array([<span style="color: #B452CD">2</span>, <span style="color: #B452CD">3</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">7</span>, <span style="color: #B452CD">22</span>*x[<span style="color: #B452CD">4</span>]])
|
||||
|
||||
<span style="color: #228B22"># Print the analytical gradient:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical gradient of f3 is: "</span>, f3_grad_analytical)
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that in this case, when sending an array as input argument, the
|
||||
output from Autograd is another array. This is the true gradient of
|
||||
the function, as opposed to the function in the previous example. By
|
||||
using arrays to represent the variables, the output from Autograd
|
||||
might be easier to work with, as the output is closer to what one
|
||||
could expect form a gradient-evaluting function.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec38">Functions using mathematical functions from Numpy </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f4</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.sqrt(<span style="color: #B452CD">1</span>+x**<span style="color: #B452CD">2</span>) + np.exp(x) + np.sin(<span style="color: #B452CD">2</span>*np.pi*x)
|
||||
|
||||
f4_grad = grad(f4)
|
||||
|
||||
x = <span style="color: #B452CD">2.7</span>
|
||||
|
||||
<span style="color: #228B22"># Print the computed derivative:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f4 at x = %g is: %g"</span>%(x,f4_grad(x)))
|
||||
|
||||
<span style="color: #228B22"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
|
||||
f4_grad_analytical = x/np.sqrt(<span style="color: #B452CD">1</span> + x**<span style="color: #B452CD">2</span>) + np.exp(x) + np.cos(<span style="color: #B452CD">2</span>*np.pi*x)*<span style="color: #B452CD">2</span>*np.pi
|
||||
|
||||
<span style="color: #228B22"># Print the analytical gradient:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical gradient of f4 at x = %g is: %g"</span>%(x,f4_grad_analytical))
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec39">More autograd </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f5</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">if</span> x >= <span style="color: #B452CD">0</span>:
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> x**<span style="color: #B452CD">2</span>
|
||||
<span style="color: #8B008B; font-weight: bold">else</span>:
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> -<span style="color: #B452CD">3</span>*x + <span style="color: #B452CD">1</span>
|
||||
|
||||
f5_grad = grad(f5)
|
||||
|
||||
x = <span style="color: #B452CD">2.7</span>
|
||||
|
||||
<span style="color: #228B22"># Print the computed derivative:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f5 at x = %g is: %g"</span>%(x,f5_grad(x)))
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec40">And with loops </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f6_for</span>(x):
|
||||
val = <span style="color: #B452CD">0</span>
|
||||
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #B452CD">10</span>):
|
||||
val = val + x**i
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> val
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f6_while</span>(x):
|
||||
val = <span style="color: #B452CD">0</span>
|
||||
i = <span style="color: #B452CD">0</span>
|
||||
<span style="color: #8B008B; font-weight: bold">while</span> i < <span style="color: #B452CD">10</span>:
|
||||
val = val + x**i
|
||||
i = i + <span style="color: #B452CD">1</span>
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> val
|
||||
|
||||
f6_for_grad = grad(f6_for)
|
||||
f6_while_grad = grad(f6_while)
|
||||
|
||||
x = <span style="color: #B452CD">0.5</span>
|
||||
|
||||
<span style="color: #228B22"># Print the computed derivaties of f6_for and f6_while</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f6_for at x = %g is: %g"</span>%(x,f6_for_grad(x)))
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f6_while at x = %g is: %g"</span>%(x,f6_while_grad(x)))
|
||||
</pre></div>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #228B22"># Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9</span>
|
||||
<span style="color: #228B22"># The analytical derivative is: sum(i*x**(i-1)) </span>
|
||||
f6_grad_analytical = <span style="color: #B452CD">0</span>
|
||||
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #B452CD">10</span>):
|
||||
f6_grad_analytical += i*x**(i-<span style="color: #B452CD">1</span>)
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical derivative of f6 at x = %g is: %g"</span>%(x,f6_grad_analytical))
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec41">Using recursion </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f7</span>(n): <span style="color: #228B22"># Assume that n is an integer</span>
|
||||
<span style="color: #8B008B; font-weight: bold">if</span> n == <span style="color: #B452CD">1</span> <span style="color: #8B008B">or</span> n == <span style="color: #B452CD">0</span>:
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">1</span>
|
||||
<span style="color: #8B008B; font-weight: bold">else</span>:
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> n*f7(n-<span style="color: #B452CD">1</span>)
|
||||
|
||||
f7_grad = grad(f7)
|
||||
|
||||
n = <span style="color: #B452CD">2.0</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f7 at n = %d is: %g"</span>%(n,f7_grad(n)))
|
||||
|
||||
<span style="color: #228B22"># The function f7 is an implementation of the factorial of n.</span>
|
||||
<span style="color: #228B22"># By using the product rule, one can find that the derivative is:</span>
|
||||
|
||||
f7_grad_analytical = <span style="color: #B452CD">0</span>
|
||||
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">int</span>(n)-<span style="color: #B452CD">1</span>):
|
||||
tmp = <span style="color: #B452CD">1</span>
|
||||
<span style="color: #8B008B; font-weight: bold">for</span> k <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">int</span>(n)-<span style="color: #B452CD">1</span>):
|
||||
<span style="color: #8B008B; font-weight: bold">if</span> k != i:
|
||||
tmp *= (n - k)
|
||||
f7_grad_analytical += tmp
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical derivative of f7 at n = %d is: %g"</span>%(n,f7_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec42">Unsupported functions </h2>
|
||||
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
|
||||
|
||||
<p>
|
||||
Assigning a value to the variable being differentiated with respect to
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f8</span>(x): <span style="color: #228B22"># Assume x is an array</span>
|
||||
x[<span style="color: #B452CD">2</span>] = <span style="color: #B452CD">3</span>
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> x*<span style="color: #B452CD">2</span>
|
||||
|
||||
f8_grad = grad(f8)
|
||||
|
||||
x = <span style="color: #B452CD">8.4</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The derivative of f8 is:"</span>,f8_grad(x))
|
||||
</pre></div>
|
||||
<p>
|
||||
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec43">The syntax a.dot(b) when finding the dot product </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f9</span>(a): <span style="color: #228B22"># Assume a is an array with 2 elements</span>
|
||||
b = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>])
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> a.dot(b)
|
||||
|
||||
f9_grad = grad(f9)
|
||||
|
||||
x = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0.0</span>])
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The derivative of f9 is:"</span>,f9_grad(x))
|
||||
</pre></div>
|
||||
<p>
|
||||
Here we are told that the 'dot' function does not belong to Autograd's
|
||||
version of a Numpy array. To overcome this, an alternative syntax
|
||||
which also computed the dot product can be used:
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f9_alternative</span>(x): <span style="color: #228B22"># Assume a is an array with 2 elements</span>
|
||||
b = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>])
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.dot(x,b) <span style="color: #228B22"># The same as x_1*b_1 + x_2*b_2</span>
|
||||
|
||||
f9_alternative_grad = grad(f9_alternative)
|
||||
|
||||
x = np.array([<span style="color: #B452CD">3.0</span>,<span style="color: #B452CD">0.0</span>])
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The gradient of f9 is:"</span>,f9_alternative_grad(x))
|
||||
|
||||
<span style="color: #228B22"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
|
||||
<span style="color: #228B22"># w.r.t x is (b_1, b_2).</span>
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec44">Recommended to avoid </h2>
|
||||
The documentation recommends to avoid inplace operations such as
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>a += b
|
||||
a -= b
|
||||
a*= b
|
||||
a /=b
|
||||
</pre></div>
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec45">Stochastic Gradient Descent </h2>
|
||||
|
||||
<p>
|
||||
Stochastic gradient descent (SGD) and variants thereof address some of
|
||||
@@ -1200,7 +1662,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec35">Computation of gradients </h2>
|
||||
<h2 id="___sec46">Computation of gradients </h2>
|
||||
|
||||
<p>
|
||||
This in turn means that the gradient can be
|
||||
@@ -1222,7 +1684,7 @@ minibatches. We denote these minibatches by \( B_k \) where
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec36">SGD example </h2>
|
||||
<h2 id="___sec47">SGD example </h2>
|
||||
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
|
||||
and we choose to have \( M=5 \) minibathces,
|
||||
then each minibatch contains two data points. In particular we have
|
||||
@@ -1248,7 +1710,7 @@ $$
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec37">The gradient step </h2>
|
||||
<h2 id="___sec48">The gradient step </h2>
|
||||
|
||||
<p>
|
||||
Thus a gradient descent step now looks like
|
||||
@@ -1269,7 +1731,7 @@ the number of minibatches, as exemplified in the code below.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec38">Simple example code </h2>
|
||||
<h2 id="___sec49">Simple example code </h2>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -1301,7 +1763,7 @@ all \( n \) datapoints.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec39">When do we stop? </h2>
|
||||
<h2 id="___sec50">When do we stop? </h2>
|
||||
|
||||
<p>
|
||||
A natural question is when do we stop the search for a new minimum?
|
||||
@@ -1318,7 +1780,7 @@ gave the lowest value.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec40">Slightly different approach </h2>
|
||||
<h2 id="___sec51">Slightly different approach </h2>
|
||||
|
||||
<p>
|
||||
Another approach is to let the step length \( \gamma_j \) depend on the
|
||||
|
||||
@@ -90,16 +90,16 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -111,13 +111,34 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -767,12 +788,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec23">The Steepest descent algorithm </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec24">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
|
||||
<h2 id="___sec23">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -812,7 +828,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">The routine for the steepest descent method </h2>
|
||||
<h2 id="___sec24">The routine for the steepest descent method </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -845,6 +861,75 @@ $$
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">Steepest descent example </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy.linalg</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">la</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">scipy.optimize</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sopt</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pt</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">mpl_toolkits.mplot3d</span> <span style="color: #8B008B; font-weight: bold">import</span> axes3d
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">0.5</span>*x[<span style="color: #B452CD">0</span>]**<span style="color: #B452CD">2</span> + <span style="color: #B452CD">2.5</span>*x[<span style="color: #B452CD">1</span>]**<span style="color: #B452CD">2</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">df</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.array([x[<span style="color: #B452CD">0</span>], <span style="color: #B452CD">5</span>*x[<span style="color: #B452CD">1</span>]])
|
||||
|
||||
fig = pt.figure()
|
||||
ax = fig.gca(projection=<span style="color: #CD5555">"3d"</span>)
|
||||
|
||||
xmesh, ymesh = np.mgrid[-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50j</span>,-<span style="color: #B452CD">2</span>:<span style="color: #B452CD">2</span>:<span style="color: #B452CD">50j</span>]
|
||||
fmesh = f(np.array([xmesh, ymesh]))
|
||||
ax.plot_surface(xmesh, ymesh, fmesh)
|
||||
</pre></div>
|
||||
<p>
|
||||
And then as countor plot
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>pt.axis(<span style="color: #CD5555">"equal"</span>)
|
||||
pt.contour(xmesh, ymesh, fmesh)
|
||||
guesses = [np.array([<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2.</span>/<span style="color: #B452CD">5</span>])]
|
||||
</pre></div>
|
||||
<p>
|
||||
Find guesses
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>x = guesses[-<span style="color: #B452CD">1</span>]
|
||||
s = -df(x)
|
||||
</pre></div>
|
||||
<p>
|
||||
Run it!
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1d</span>(alpha):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> f(x + alpha*s)
|
||||
|
||||
alpha_opt = sopt.golden(f1d)
|
||||
next_guess = x + alpha_opt * s
|
||||
guesses.append(next_guess)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(next_guess)
|
||||
</pre></div>
|
||||
<p>
|
||||
What happened?
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>pt.axis(<span style="color: #CD5555">"equal"</span>)
|
||||
pt.contour(xmesh, ymesh, fmesh, <span style="color: #B452CD">50</span>)
|
||||
it_array = np.array(guesses)
|
||||
pt.plot(it_array.T[<span style="color: #B452CD">0</span>], it_array.T[<span style="color: #B452CD">1</span>], <span style="color: #CD5555">"x-"</span>)
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
@@ -1106,7 +1191,393 @@ beta_ridge = np.dot(Z,np.dot(X.T,y))
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec34">Stochastic Gradient Descent </h2>
|
||||
<h2 id="___sec34">Automatic differentiation </h2>
|
||||
Python has tools for so-called <b>automatic differentiation</b>.
|
||||
Consider the following example
|
||||
$$
|
||||
f(x) = \sin\left(2\pi x + x^2\right)
|
||||
$$
|
||||
|
||||
which has the following derivative
|
||||
$$
|
||||
f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right)
|
||||
$$
|
||||
|
||||
Using <b>autograd</b> we have
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
|
||||
<span style="color: #228B22"># To do elementwise differentiation:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> elementwise_grad <span style="color: #8B008B; font-weight: bold">as</span> egrad
|
||||
|
||||
<span style="color: #228B22"># To plot:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
||||
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.sin(<span style="color: #B452CD">2</span>*np.pi*x + x**<span style="color: #B452CD">2</span>)
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f_grad_analytic</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.cos(<span style="color: #B452CD">2</span>*np.pi*x + x**<span style="color: #B452CD">2</span>)*(<span style="color: #B452CD">2</span>*np.pi + <span style="color: #B452CD">2</span>*x)
|
||||
|
||||
<span style="color: #228B22"># Do the comparison:</span>
|
||||
x = np.linspace(<span style="color: #B452CD">0</span>,<span style="color: #B452CD">1</span>,<span style="color: #B452CD">1000</span>)
|
||||
|
||||
f_grad = egrad(f)
|
||||
|
||||
computed = f_grad(x)
|
||||
analytic = f_grad_analytic(x)
|
||||
|
||||
plt.title(<span style="color: #CD5555">'Derivative computed from Autograd compared with the analytical derivative'</span>)
|
||||
plt.plot(x,computed,label=<span style="color: #CD5555">'autograd'</span>)
|
||||
plt.plot(x,analytic,label=<span style="color: #CD5555">'analytic'</span>)
|
||||
|
||||
plt.xlabel(<span style="color: #CD5555">'x'</span>)
|
||||
plt.ylabel(<span style="color: #CD5555">'y'</span>)
|
||||
plt.legend()
|
||||
|
||||
plt.show()
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The max absolute difference is: %g"</span>%(np.max(np.abs(computed - analytic))))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec35">Using autograd </h2>
|
||||
|
||||
<p>
|
||||
Here we
|
||||
experiment with what kind of functions Autograd is capable
|
||||
of finding the gradient of. The following Python functions are just
|
||||
meant to illustrate what Autograd can do, but please feel free to
|
||||
experiment with other, possibly more complicated, functions as well.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> x**<span style="color: #B452CD">3</span> + <span style="color: #B452CD">1</span>
|
||||
|
||||
f1_grad = grad(f1)
|
||||
|
||||
<span style="color: #228B22"># Remember to send in float as argument to the computed gradient from Autograd!</span>
|
||||
a = <span style="color: #B452CD">1.0</span>
|
||||
|
||||
<span style="color: #228B22"># See the evaluated gradient at a using autograd:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The gradient of f1 evaluated at a = %g using autograd is: %g"</span>%(a,f1_grad(a)))
|
||||
|
||||
<span style="color: #228B22"># Compare with the analytical derivative, that is f1'(x) = 3*x**2 </span>
|
||||
grad_analytical = <span style="color: #B452CD">3</span>*a**<span style="color: #B452CD">2</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g"</span>%(a,grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec36">Autograd with more complicated functions </h2>
|
||||
|
||||
<p>
|
||||
To differentiate with respect to two (or more) arguments of a Python
|
||||
function, Autograd need to know at which variable the function if
|
||||
being differentiated with respect to.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f2</span>(x1,x2):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">3</span>*x1**<span style="color: #B452CD">3</span> + x2*(x1 - <span style="color: #B452CD">5</span>) + <span style="color: #B452CD">1</span>
|
||||
|
||||
<span style="color: #228B22"># By sending the argument 0, Autograd will compute the derivative w.r.t the first variable, in this case x1</span>
|
||||
f2_grad_x1 = grad(f2,<span style="color: #B452CD">0</span>)
|
||||
|
||||
<span style="color: #228B22"># ... and differentiate w.r.t x2 by sending 1 as an additional arugment to grad</span>
|
||||
f2_grad_x2 = grad(f2,<span style="color: #B452CD">1</span>)
|
||||
|
||||
x1 = <span style="color: #B452CD">1.0</span>
|
||||
x2 = <span style="color: #B452CD">3.0</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"Evaluating at x1 = %g, x2 = %g"</span>%(x1,x2))
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"-"</span>*<span style="color: #B452CD">30</span>)
|
||||
|
||||
<span style="color: #228B22"># Compare with the analytical derivatives:</span>
|
||||
|
||||
<span style="color: #228B22"># Derivative of f2 w.r.t x1 is: 9*x1**2 + x2:</span>
|
||||
f2_grad_x1_analytical = <span style="color: #B452CD">9</span>*x1**<span style="color: #B452CD">2</span> + x2
|
||||
|
||||
<span style="color: #228B22"># Derivative of f2 w.r.t x2 is: x1 - 5:</span>
|
||||
f2_grad_x2_analytical = x1 - <span style="color: #B452CD">5</span>
|
||||
|
||||
<span style="color: #228B22"># See the evaluated derivations:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The derivative of f2 w.r.t x1: %g"</span>%( f2_grad_x1(x1,x2) ))
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical derivative of f2 w.r.t x1: %g"</span>%( f2_grad_x1(x1,x2) ))
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>()
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The derivative of f2 w.r.t x2: %g"</span>%( f2_grad_x2(x1,x2) ))
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical derivative of f2 w.r.t x2: %g"</span>%( f2_grad_x2(x1,x2) ))
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec37">More complicated functions using the elements of their arguments directly </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f3</span>(x): <span style="color: #228B22"># Assumes x is an array of length 5 or higher</span>
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">2</span>*x[<span style="color: #B452CD">0</span>] + <span style="color: #B452CD">3</span>*x[<span style="color: #B452CD">1</span>] + <span style="color: #B452CD">5</span>*x[<span style="color: #B452CD">2</span>] + <span style="color: #B452CD">7</span>*x[<span style="color: #B452CD">3</span>] + <span style="color: #B452CD">11</span>*x[<span style="color: #B452CD">4</span>]**<span style="color: #B452CD">2</span>
|
||||
|
||||
f3_grad = grad(f3)
|
||||
|
||||
x = np.linspace(<span style="color: #B452CD">0</span>,<span style="color: #B452CD">4</span>,<span style="color: #B452CD">5</span>)
|
||||
|
||||
<span style="color: #228B22"># Print the computed gradient:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed gradient of f3 is: "</span>, f3_grad(x))
|
||||
|
||||
<span style="color: #228B22"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
|
||||
f3_grad_analytical = np.array([<span style="color: #B452CD">2</span>, <span style="color: #B452CD">3</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">7</span>, <span style="color: #B452CD">22</span>*x[<span style="color: #B452CD">4</span>]])
|
||||
|
||||
<span style="color: #228B22"># Print the analytical gradient:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical gradient of f3 is: "</span>, f3_grad_analytical)
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that in this case, when sending an array as input argument, the
|
||||
output from Autograd is another array. This is the true gradient of
|
||||
the function, as opposed to the function in the previous example. By
|
||||
using arrays to represent the variables, the output from Autograd
|
||||
might be easier to work with, as the output is closer to what one
|
||||
could expect form a gradient-evaluting function.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec38">Functions using mathematical functions from Numpy </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f4</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.sqrt(<span style="color: #B452CD">1</span>+x**<span style="color: #B452CD">2</span>) + np.exp(x) + np.sin(<span style="color: #B452CD">2</span>*np.pi*x)
|
||||
|
||||
f4_grad = grad(f4)
|
||||
|
||||
x = <span style="color: #B452CD">2.7</span>
|
||||
|
||||
<span style="color: #228B22"># Print the computed derivative:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f4 at x = %g is: %g"</span>%(x,f4_grad(x)))
|
||||
|
||||
<span style="color: #228B22"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
|
||||
f4_grad_analytical = x/np.sqrt(<span style="color: #B452CD">1</span> + x**<span style="color: #B452CD">2</span>) + np.exp(x) + np.cos(<span style="color: #B452CD">2</span>*np.pi*x)*<span style="color: #B452CD">2</span>*np.pi
|
||||
|
||||
<span style="color: #228B22"># Print the analytical gradient:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical gradient of f4 at x = %g is: %g"</span>%(x,f4_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec39">More autograd </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f5</span>(x):
|
||||
<span style="color: #8B008B; font-weight: bold">if</span> x >= <span style="color: #B452CD">0</span>:
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> x**<span style="color: #B452CD">2</span>
|
||||
<span style="color: #8B008B; font-weight: bold">else</span>:
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> -<span style="color: #B452CD">3</span>*x + <span style="color: #B452CD">1</span>
|
||||
|
||||
f5_grad = grad(f5)
|
||||
|
||||
x = <span style="color: #B452CD">2.7</span>
|
||||
|
||||
<span style="color: #228B22"># Print the computed derivative:</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f5 at x = %g is: %g"</span>%(x,f5_grad(x)))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec40">And with loops </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f6_for</span>(x):
|
||||
val = <span style="color: #B452CD">0</span>
|
||||
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #B452CD">10</span>):
|
||||
val = val + x**i
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> val
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f6_while</span>(x):
|
||||
val = <span style="color: #B452CD">0</span>
|
||||
i = <span style="color: #B452CD">0</span>
|
||||
<span style="color: #8B008B; font-weight: bold">while</span> i < <span style="color: #B452CD">10</span>:
|
||||
val = val + x**i
|
||||
i = i + <span style="color: #B452CD">1</span>
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> val
|
||||
|
||||
f6_for_grad = grad(f6_for)
|
||||
f6_while_grad = grad(f6_while)
|
||||
|
||||
x = <span style="color: #B452CD">0.5</span>
|
||||
|
||||
<span style="color: #228B22"># Print the computed derivaties of f6_for and f6_while</span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f6_for at x = %g is: %g"</span>%(x,f6_for_grad(x)))
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f6_while at x = %g is: %g"</span>%(x,f6_while_grad(x)))
|
||||
</pre></div>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #228B22"># Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9</span>
|
||||
<span style="color: #228B22"># The analytical derivative is: sum(i*x**(i-1)) </span>
|
||||
f6_grad_analytical = <span style="color: #B452CD">0</span>
|
||||
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #B452CD">10</span>):
|
||||
f6_grad_analytical += i*x**(i-<span style="color: #B452CD">1</span>)
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical derivative of f6 at x = %g is: %g"</span>%(x,f6_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec41">Using recursion </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f7</span>(n): <span style="color: #228B22"># Assume that n is an integer</span>
|
||||
<span style="color: #8B008B; font-weight: bold">if</span> n == <span style="color: #B452CD">1</span> <span style="color: #8B008B">or</span> n == <span style="color: #B452CD">0</span>:
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">1</span>
|
||||
<span style="color: #8B008B; font-weight: bold">else</span>:
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> n*f7(n-<span style="color: #B452CD">1</span>)
|
||||
|
||||
f7_grad = grad(f7)
|
||||
|
||||
n = <span style="color: #B452CD">2.0</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The computed derivative of f7 at n = %d is: %g"</span>%(n,f7_grad(n)))
|
||||
|
||||
<span style="color: #228B22"># The function f7 is an implementation of the factorial of n.</span>
|
||||
<span style="color: #228B22"># By using the product rule, one can find that the derivative is:</span>
|
||||
|
||||
f7_grad_analytical = <span style="color: #B452CD">0</span>
|
||||
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">int</span>(n)-<span style="color: #B452CD">1</span>):
|
||||
tmp = <span style="color: #B452CD">1</span>
|
||||
<span style="color: #8B008B; font-weight: bold">for</span> k <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">int</span>(n)-<span style="color: #B452CD">1</span>):
|
||||
<span style="color: #8B008B; font-weight: bold">if</span> k != i:
|
||||
tmp *= (n - k)
|
||||
f7_grad_analytical += tmp
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The analytical derivative of f7 at n = %d is: %g"</span>%(n,f7_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec42">Unsupported functions </h2>
|
||||
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
|
||||
|
||||
<p>
|
||||
Assigning a value to the variable being differentiated with respect to
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f8</span>(x): <span style="color: #228B22"># Assume x is an array</span>
|
||||
x[<span style="color: #B452CD">2</span>] = <span style="color: #B452CD">3</span>
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> x*<span style="color: #B452CD">2</span>
|
||||
|
||||
f8_grad = grad(f8)
|
||||
|
||||
x = <span style="color: #B452CD">8.4</span>
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The derivative of f8 is:"</span>,f8_grad(x))
|
||||
</pre></div>
|
||||
<p>
|
||||
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec43">The syntax a.dot(b) when finding the dot product </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f9</span>(a): <span style="color: #228B22"># Assume a is an array with 2 elements</span>
|
||||
b = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>])
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> a.dot(b)
|
||||
|
||||
f9_grad = grad(f9)
|
||||
|
||||
x = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0.0</span>])
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The derivative of f9 is:"</span>,f9_grad(x))
|
||||
</pre></div>
|
||||
<p>
|
||||
Here we are told that the 'dot' function does not belong to Autograd's
|
||||
version of a Numpy array. To overcome this, an alternative syntax
|
||||
which also computed the dot product can be used:
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">autograd.numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">autograd</span> <span style="color: #8B008B; font-weight: bold">import</span> grad
|
||||
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f9_alternative</span>(x): <span style="color: #228B22"># Assume a is an array with 2 elements</span>
|
||||
b = np.array([<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">2.0</span>])
|
||||
<span style="color: #8B008B; font-weight: bold">return</span> np.dot(x,b) <span style="color: #228B22"># The same as x_1*b_1 + x_2*b_2</span>
|
||||
|
||||
f9_alternative_grad = grad(f9_alternative)
|
||||
|
||||
x = np.array([<span style="color: #B452CD">3.0</span>,<span style="color: #B452CD">0.0</span>])
|
||||
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"The gradient of f9 is:"</span>,f9_alternative_grad(x))
|
||||
|
||||
<span style="color: #228B22"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
|
||||
<span style="color: #228B22"># w.r.t x is (b_1, b_2).</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec44">Recommended to avoid </h2>
|
||||
The documentation recommends to avoid inplace operations such as
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>a += b
|
||||
a -= b
|
||||
a*= b
|
||||
a /=b
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec45">Stochastic Gradient Descent </h2>
|
||||
|
||||
<p>
|
||||
Stochastic gradient descent (SGD) and variants thereof address some of
|
||||
@@ -1124,7 +1595,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec35">Computation of gradients </h2>
|
||||
<h2 id="___sec46">Computation of gradients </h2>
|
||||
|
||||
<p>
|
||||
This in turn means that the gradient can be
|
||||
@@ -1144,7 +1615,7 @@ minibatches. We denote these minibatches by \( B_k \) where
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec36">SGD example </h2>
|
||||
<h2 id="___sec47">SGD example </h2>
|
||||
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
|
||||
and we choose to have \( M=5 \) minibathces,
|
||||
then each minibatch contains two data points. In particular we have
|
||||
@@ -1168,7 +1639,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec37">The gradient step </h2>
|
||||
<h2 id="___sec48">The gradient step </h2>
|
||||
|
||||
<p>
|
||||
Thus a gradient descent step now looks like
|
||||
@@ -1187,7 +1658,7 @@ the number of minibatches, as exemplified in the code below.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec38">Simple example code </h2>
|
||||
<h2 id="___sec49">Simple example code </h2>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -1219,7 +1690,7 @@ all \( n \) datapoints.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec39">When do we stop? </h2>
|
||||
<h2 id="___sec50">When do we stop? </h2>
|
||||
|
||||
<p>
|
||||
A natural question is when do we stop the search for a new minimum?
|
||||
@@ -1236,7 +1707,7 @@ gave the lowest value.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec40">Slightly different approach </h2>
|
||||
<h2 id="___sec51">Slightly different approach </h2>
|
||||
|
||||
<p>
|
||||
Another approach is to let the step length \( \gamma_j \) depend on the
|
||||
|
||||
@@ -95,16 +95,16 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
('Steepest descent method', 2, None, '___sec20'),
|
||||
('Gradient descent method', 2, None, '___sec21'),
|
||||
('Final expressions', 2, None, '___sec22'),
|
||||
('The Steepest descent algorithm', 2, None, '___sec23'),
|
||||
('Simple codes for steepest descent and conjugate gradient '
|
||||
'using a $2\\times 2$ matrix, in c++, Python code to come',
|
||||
2,
|
||||
None,
|
||||
'___sec24'),
|
||||
'___sec23'),
|
||||
('The routine for the steepest descent method',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
'___sec24'),
|
||||
('Steepest descent example', 2, None, '___sec25'),
|
||||
('Revisiting our first homework', 2, None, '___sec26'),
|
||||
('Gradient descent example', 2, None, '___sec27'),
|
||||
('The derivative of the cost/loss function', 2, None, '___sec28'),
|
||||
@@ -116,13 +116,34 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
None,
|
||||
'___sec32'),
|
||||
('Gradient descent and Ridge', 2, None, '___sec33'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec34'),
|
||||
('Computation of gradients', 2, None, '___sec35'),
|
||||
('SGD example', 2, None, '___sec36'),
|
||||
('The gradient step', 2, None, '___sec37'),
|
||||
('Simple example code', 2, None, '___sec38'),
|
||||
('When do we stop?', 2, None, '___sec39'),
|
||||
('Slightly different approach', 2, None, '___sec40')]}
|
||||
('Automatic differentiation', 2, None, '___sec34'),
|
||||
('Using autograd', 2, None, '___sec35'),
|
||||
('Autograd with more complicated functions', 2, None, '___sec36'),
|
||||
('More complicated functions using the elements of their '
|
||||
'arguments directly',
|
||||
2,
|
||||
None,
|
||||
'___sec37'),
|
||||
('Functions using mathematical functions from Numpy',
|
||||
2,
|
||||
None,
|
||||
'___sec38'),
|
||||
('More autograd', 2, None, '___sec39'),
|
||||
('And with loops', 2, None, '___sec40'),
|
||||
('Using recursion', 2, None, '___sec41'),
|
||||
('Unsupported functions', 2, None, '___sec42'),
|
||||
('The syntax a.dot(b) when finding the dot product',
|
||||
2,
|
||||
None,
|
||||
'___sec43'),
|
||||
('Recommended to avoid', 2, None, '___sec44'),
|
||||
('Stochastic Gradient Descent', 2, None, '___sec45'),
|
||||
('Computation of gradients', 2, None, '___sec46'),
|
||||
('SGD example', 2, None, '___sec47'),
|
||||
('The gradient step', 2, None, '___sec48'),
|
||||
('Simple example code', 2, None, '___sec49'),
|
||||
('When do we stop?', 2, None, '___sec50'),
|
||||
('Slightly different approach', 2, None, '___sec51')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -772,12 +793,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec23">The Steepest descent algorithm </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec24">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
|
||||
<h2 id="___sec23">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -817,7 +833,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">The routine for the steepest descent method </h2>
|
||||
<h2 id="___sec24">The routine for the steepest descent method </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -850,6 +866,75 @@ $$
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">Steepest descent example </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy.linalg</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">la</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scipy.optimize</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sopt</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pt</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> axes3d
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">0.5*</span>x[<span style="color: #666666">0</span>]<span style="color: #666666">**2</span> <span style="color: #666666">+</span> <span style="color: #666666">2.5*</span>x[<span style="color: #666666">1</span>]<span style="color: #666666">**2</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">df</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>array([x[<span style="color: #666666">0</span>], <span style="color: #666666">5*</span>x[<span style="color: #666666">1</span>]])
|
||||
|
||||
fig <span style="color: #666666">=</span> pt<span style="color: #666666">.</span>figure()
|
||||
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>gca(projection<span style="color: #666666">=</span><span style="color: #BA2121">"3d"</span>)
|
||||
|
||||
xmesh, ymesh <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mgrid[<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>,<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>]
|
||||
fmesh <span style="color: #666666">=</span> f(np<span style="color: #666666">.</span>array([xmesh, ymesh]))
|
||||
ax<span style="color: #666666">.</span>plot_surface(xmesh, ymesh, fmesh)
|
||||
</pre></div>
|
||||
<p>
|
||||
And then as countor plot
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">"equal"</span>)
|
||||
pt<span style="color: #666666">.</span>contour(xmesh, ymesh, fmesh)
|
||||
guesses <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">2./5</span>])]
|
||||
</pre></div>
|
||||
<p>
|
||||
Find guesses
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> guesses[<span style="color: #666666">-1</span>]
|
||||
s <span style="color: #666666">=</span> <span style="color: #666666">-</span>df(x)
|
||||
</pre></div>
|
||||
<p>
|
||||
Run it!
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1d</span>(alpha):
|
||||
<span style="color: #008000; font-weight: bold">return</span> f(x <span style="color: #666666">+</span> alpha<span style="color: #666666">*</span>s)
|
||||
|
||||
alpha_opt <span style="color: #666666">=</span> sopt<span style="color: #666666">.</span>golden(f1d)
|
||||
next_guess <span style="color: #666666">=</span> x <span style="color: #666666">+</span> alpha_opt <span style="color: #666666">*</span> s
|
||||
guesses<span style="color: #666666">.</span>append(next_guess)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(next_guess)
|
||||
</pre></div>
|
||||
<p>
|
||||
What happened?
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">"equal"</span>)
|
||||
pt<span style="color: #666666">.</span>contour(xmesh, ymesh, fmesh, <span style="color: #666666">50</span>)
|
||||
it_array <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array(guesses)
|
||||
pt<span style="color: #666666">.</span>plot(it_array<span style="color: #666666">.</span>T[<span style="color: #666666">0</span>], it_array<span style="color: #666666">.</span>T[<span style="color: #666666">1</span>], <span style="color: #BA2121">"x-"</span>)
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
@@ -1111,7 +1196,393 @@ beta_ridge <span style="color: #666666">=</span> np<span style="color: #666666">
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec34">Stochastic Gradient Descent </h2>
|
||||
<h2 id="___sec34">Automatic differentiation </h2>
|
||||
Python has tools for so-called <b>automatic differentiation</b>.
|
||||
Consider the following example
|
||||
$$
|
||||
f(x) = \sin\left(2\pi x + x^2\right)
|
||||
$$
|
||||
|
||||
which has the following derivative
|
||||
$$
|
||||
f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right)
|
||||
$$
|
||||
|
||||
Using <b>autograd</b> we have
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># To do elementwise differentiation:</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> elementwise_grad <span style="color: #008000; font-weight: bold">as</span> egrad
|
||||
|
||||
<span style="color: #408080; font-style: italic"># To plot:</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>sin(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f_grad_analytic</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>cos(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>)<span style="color: #666666">*</span>(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi <span style="color: #666666">+</span> <span style="color: #666666">2*</span>x)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Do the comparison:</span>
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">0</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1000</span>)
|
||||
|
||||
f_grad <span style="color: #666666">=</span> egrad(f)
|
||||
|
||||
computed <span style="color: #666666">=</span> f_grad(x)
|
||||
analytic <span style="color: #666666">=</span> f_grad_analytic(x)
|
||||
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">'Derivative computed from Autograd compared with the analytical derivative'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x,computed,label<span style="color: #666666">=</span><span style="color: #BA2121">'autograd'</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x,analytic,label<span style="color: #666666">=</span><span style="color: #BA2121">'analytic'</span>)
|
||||
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">'x'</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">'y'</span>)
|
||||
plt<span style="color: #666666">.</span>legend()
|
||||
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The max absolute difference is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(np<span style="color: #666666">.</span>max(np<span style="color: #666666">.</span>abs(computed <span style="color: #666666">-</span> analytic))))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec35">Using autograd </h2>
|
||||
|
||||
<p>
|
||||
Here we
|
||||
experiment with what kind of functions Autograd is capable
|
||||
of finding the gradient of. The following Python functions are just
|
||||
meant to illustrate what Autograd can do, but please feel free to
|
||||
experiment with other, possibly more complicated, functions as well.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">**3</span> <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
|
||||
f1_grad <span style="color: #666666">=</span> grad(f1)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Remember to send in float as argument to the computed gradient from Autograd!</span>
|
||||
a <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># See the evaluated gradient at a using autograd:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> using autograd is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(a,f1_grad(a)))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Compare with the analytical derivative, that is f1'(x) = 3*x**2 </span>
|
||||
grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">3*</span>a<span style="color: #666666">**2</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> by finding the analytic expression is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(a,grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec36">Autograd with more complicated functions </h2>
|
||||
|
||||
<p>
|
||||
To differentiate with respect to two (or more) arguments of a Python
|
||||
function, Autograd need to know at which variable the function if
|
||||
being differentiated with respect to.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f2</span>(x1,x2):
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">3*</span>x1<span style="color: #666666">**3</span> <span style="color: #666666">+</span> x2<span style="color: #666666">*</span>(x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># By sending the argument 0, Autograd will compute the derivative w.r.t the first variable, in this case x1</span>
|
||||
f2_grad_x1 <span style="color: #666666">=</span> grad(f2,<span style="color: #666666">0</span>)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># ... and differentiate w.r.t x2 by sending 1 as an additional arugment to grad</span>
|
||||
f2_grad_x2 <span style="color: #666666">=</span> grad(f2,<span style="color: #666666">1</span>)
|
||||
|
||||
x1 <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
|
||||
x2 <span style="color: #666666">=</span> <span style="color: #666666">3.0</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Evaluating at x1 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">, x2 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x1,x2))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"-"</span><span style="color: #666666">*30</span>)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Compare with the analytical derivatives:</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Derivative of f2 w.r.t x1 is: 9*x1**2 + x2:</span>
|
||||
f2_grad_x1_analytical <span style="color: #666666">=</span> <span style="color: #666666">9*</span>x1<span style="color: #666666">**2</span> <span style="color: #666666">+</span> x2
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Derivative of f2 w.r.t x2 is: x1 - 5:</span>
|
||||
f2_grad_x2_analytical <span style="color: #666666">=</span> x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># See the evaluated derivations:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>()
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec37">More complicated functions using the elements of their arguments directly </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f3</span>(x): <span style="color: #408080; font-style: italic"># Assumes x is an array of length 5 or higher</span>
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">2*</span>x[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> <span style="color: #666666">3*</span>x[<span style="color: #666666">1</span>] <span style="color: #666666">+</span> <span style="color: #666666">5*</span>x[<span style="color: #666666">2</span>] <span style="color: #666666">+</span> <span style="color: #666666">7*</span>x[<span style="color: #666666">3</span>] <span style="color: #666666">+</span> <span style="color: #666666">11*</span>x[<span style="color: #666666">4</span>]<span style="color: #666666">**2</span>
|
||||
|
||||
f3_grad <span style="color: #666666">=</span> grad(f3)
|
||||
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">0</span>,<span style="color: #666666">4</span>,<span style="color: #666666">5</span>)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the computed gradient:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed gradient of f3 is: "</span>, f3_grad(x))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
|
||||
f3_grad_analytical <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">5</span>, <span style="color: #666666">7</span>, <span style="color: #666666">22*</span>x[<span style="color: #666666">4</span>]])
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical gradient of f3 is: "</span>, f3_grad_analytical)
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that in this case, when sending an array as input argument, the
|
||||
output from Autograd is another array. This is the true gradient of
|
||||
the function, as opposed to the function in the previous example. By
|
||||
using arrays to represent the variables, the output from Autograd
|
||||
might be easier to work with, as the output is closer to what one
|
||||
could expect form a gradient-evaluting function.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec38">Functions using mathematical functions from Numpy </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f4</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1+</span>x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>sin(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)
|
||||
|
||||
f4_grad <span style="color: #666666">=</span> grad(f4)
|
||||
|
||||
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f4_grad(x)))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
|
||||
f4_grad_analytical <span style="color: #666666">=</span> x<span style="color: #666666">/</span>np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1</span> <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>cos(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)<span style="color: #666666">*2*</span>np<span style="color: #666666">.</span>pi
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical gradient of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f4_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec39">More autograd </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f5</span>(x):
|
||||
<span style="color: #008000; font-weight: bold">if</span> x <span style="color: #666666">>=</span> <span style="color: #666666">0</span>:
|
||||
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">**2</span>
|
||||
<span style="color: #008000; font-weight: bold">else</span>:
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">-3*</span>x <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
|
||||
f5_grad <span style="color: #666666">=</span> grad(f5)
|
||||
|
||||
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f5 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f5_grad(x)))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec40">And with loops </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f6_for</span>(x):
|
||||
val <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10</span>):
|
||||
val <span style="color: #666666">=</span> val <span style="color: #666666">+</span> x<span style="color: #666666">**</span>i
|
||||
<span style="color: #008000; font-weight: bold">return</span> val
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f6_while</span>(x):
|
||||
val <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
i <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">while</span> i <span style="color: #666666"><</span> <span style="color: #666666">10</span>:
|
||||
val <span style="color: #666666">=</span> val <span style="color: #666666">+</span> x<span style="color: #666666">**</span>i
|
||||
i <span style="color: #666666">=</span> i <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
<span style="color: #008000; font-weight: bold">return</span> val
|
||||
|
||||
f6_for_grad <span style="color: #666666">=</span> grad(f6_for)
|
||||
f6_while_grad <span style="color: #666666">=</span> grad(f6_while)
|
||||
|
||||
x <span style="color: #666666">=</span> <span style="color: #666666">0.5</span>
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Print the computed derivaties of f6_for and f6_while</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f6_for at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f6_for_grad(x)))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f6_while at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f6_while_grad(x)))
|
||||
</pre></div>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #408080; font-style: italic"># Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9</span>
|
||||
<span style="color: #408080; font-style: italic"># The analytical derivative is: sum(i*x**(i-1)) </span>
|
||||
f6_grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10</span>):
|
||||
f6_grad_analytical <span style="color: #666666">+=</span> i<span style="color: #666666">*</span>x<span style="color: #666666">**</span>(i<span style="color: #666666">-1</span>)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical derivative of f6 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(x,f6_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec41">Using recursion </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f7</span>(n): <span style="color: #408080; font-style: italic"># Assume that n is an integer</span>
|
||||
<span style="color: #008000; font-weight: bold">if</span> n <span style="color: #666666">==</span> <span style="color: #666666">1</span> <span style="color: #AA22FF; font-weight: bold">or</span> n <span style="color: #666666">==</span> <span style="color: #666666">0</span>:
|
||||
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1</span>
|
||||
<span style="color: #008000; font-weight: bold">else</span>:
|
||||
<span style="color: #008000; font-weight: bold">return</span> n<span style="color: #666666">*</span>f7(n<span style="color: #666666">-1</span>)
|
||||
|
||||
f7_grad <span style="color: #666666">=</span> grad(f7)
|
||||
|
||||
n <span style="color: #666666">=</span> <span style="color: #666666">2.0</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The computed derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(n,f7_grad(n)))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># The function f7 is an implementation of the factorial of n.</span>
|
||||
<span style="color: #408080; font-style: italic"># By using the product rule, one can find that the derivative is:</span>
|
||||
|
||||
f7_grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">0</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">int</span>(n)<span style="color: #666666">-1</span>):
|
||||
tmp <span style="color: #666666">=</span> <span style="color: #666666">1</span>
|
||||
<span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">int</span>(n)<span style="color: #666666">-1</span>):
|
||||
<span style="color: #008000; font-weight: bold">if</span> k <span style="color: #666666">!=</span> i:
|
||||
tmp <span style="color: #666666">*=</span> (n <span style="color: #666666">-</span> k)
|
||||
f7_grad_analytical <span style="color: #666666">+=</span> tmp
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The analytical derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">"</span><span style="color: #666666">%</span>(n,f7_grad_analytical))
|
||||
</pre></div>
|
||||
<p>
|
||||
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec42">Unsupported functions </h2>
|
||||
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
|
||||
|
||||
<p>
|
||||
Assigning a value to the variable being differentiated with respect to
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f8</span>(x): <span style="color: #408080; font-style: italic"># Assume x is an array</span>
|
||||
x[<span style="color: #666666">2</span>] <span style="color: #666666">=</span> <span style="color: #666666">3</span>
|
||||
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">*2</span>
|
||||
|
||||
f8_grad <span style="color: #666666">=</span> grad(f8)
|
||||
|
||||
x <span style="color: #666666">=</span> <span style="color: #666666">8.4</span>
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The derivative of f8 is:"</span>,f8_grad(x))
|
||||
</pre></div>
|
||||
<p>
|
||||
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec43">The syntax a.dot(b) when finding the dot product </h2>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f9</span>(a): <span style="color: #408080; font-style: italic"># Assume a is an array with 2 elements</span>
|
||||
b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">2.0</span>])
|
||||
<span style="color: #008000; font-weight: bold">return</span> a<span style="color: #666666">.</span>dot(b)
|
||||
|
||||
f9_grad <span style="color: #666666">=</span> grad(f9)
|
||||
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">0.0</span>])
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The derivative of f9 is:"</span>,f9_grad(x))
|
||||
</pre></div>
|
||||
<p>
|
||||
Here we are told that the 'dot' function does not belong to Autograd's
|
||||
version of a Numpy array. To overcome this, an alternative syntax
|
||||
which also computed the dot product can be used:
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f9_alternative</span>(x): <span style="color: #408080; font-style: italic"># Assume a is an array with 2 elements</span>
|
||||
b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">2.0</span>])
|
||||
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>dot(x,b) <span style="color: #408080; font-style: italic"># The same as x_1*b_1 + x_2*b_2</span>
|
||||
|
||||
f9_alternative_grad <span style="color: #666666">=</span> grad(f9_alternative)
|
||||
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">3.0</span>,<span style="color: #666666">0.0</span>])
|
||||
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"The gradient of f9 is:"</span>,f9_alternative_grad(x))
|
||||
|
||||
<span style="color: #408080; font-style: italic"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
|
||||
<span style="color: #408080; font-style: italic"># w.r.t x is (b_1, b_2).</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec44">Recommended to avoid </h2>
|
||||
The documentation recommends to avoid inplace operations such as
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>a <span style="color: #666666">+=</span> b
|
||||
a <span style="color: #666666">-=</span> b
|
||||
a<span style="color: #666666">*=</span> b
|
||||
a <span style="color: #666666">/=</span>b
|
||||
</pre></div>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec45">Stochastic Gradient Descent </h2>
|
||||
|
||||
<p>
|
||||
Stochastic gradient descent (SGD) and variants thereof address some of
|
||||
@@ -1129,7 +1600,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec35">Computation of gradients </h2>
|
||||
<h2 id="___sec46">Computation of gradients </h2>
|
||||
|
||||
<p>
|
||||
This in turn means that the gradient can be
|
||||
@@ -1149,7 +1620,7 @@ minibatches. We denote these minibatches by \( B_k \) where
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec36">SGD example </h2>
|
||||
<h2 id="___sec47">SGD example </h2>
|
||||
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
|
||||
and we choose to have \( M=5 \) minibathces,
|
||||
then each minibatch contains two data points. In particular we have
|
||||
@@ -1173,7 +1644,7 @@ $$
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec37">The gradient step </h2>
|
||||
<h2 id="___sec48">The gradient step </h2>
|
||||
|
||||
<p>
|
||||
Thus a gradient descent step now looks like
|
||||
@@ -1192,7 +1663,7 @@ the number of minibatches, as exemplified in the code below.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec38">Simple example code </h2>
|
||||
<h2 id="___sec49">Simple example code </h2>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -1224,7 +1695,7 @@ all \( n \) datapoints.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec39">When do we stop? </h2>
|
||||
<h2 id="___sec50">When do we stop? </h2>
|
||||
|
||||
<p>
|
||||
A natural question is when do we stop the search for a new minimum?
|
||||
@@ -1241,7 +1712,7 @@ gave the lowest value.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec40">Slightly different approach </h2>
|
||||
<h2 id="___sec51">Slightly different approach </h2>
|
||||
|
||||
<p>
|
||||
Another approach is to let the step length \( \gamma_j \) depend on the
|
||||
|
||||
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user