lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week39

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-25 22:57:46 +02:00
parent 320ce4a5dd
commit 7a5f4e8279
7 changed files with 688 additions and 501 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/week39-bs.html
+172 -109
View File
@@ -1,11 +1,11 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 39: Optimization and Gradient Methods">
@@ -41,74 +41,137 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 39', 2, None, '___sec0'),
('Thursday September 24', 2, None, '___sec1'),
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
('Thursday September 30', 2, None, 'thursday-september-30'),
('Optimization, the central part of any Machine Learning '
'algortithm',
2,
None,
'___sec2'),
('Revisiting our Logistic Regression case', 2, None, '___sec3'),
('The equations to solve', 2, None, '___sec4'),
("Solving using Newton-Raphson's method", 2, None, '___sec5'),
("Brief reminder on Newton-Raphson's method", 2, None, '___sec6'),
('The equations', 2, None, '___sec7'),
('Simple geometric interpretation', 2, None, '___sec8'),
('Extending to more than one variable', 2, None, '___sec9'),
('Steepest descent', 2, None, '___sec10'),
('More on Steepest descent', 2, None, '___sec11'),
('The ideal', 2, None, '___sec12'),
'optimization-the-central-part-of-any-machine-learning-algortithm'),
('Revisiting our Logistic Regression case',
2,
None,
'revisiting-our-logistic-regression-case'),
('The equations to solve', 2, None, 'the-equations-to-solve'),
("Solving using Newton-Raphson's method",
2,
None,
'solving-using-newton-raphson-s-method'),
("Brief reminder on Newton-Raphson's method",
2,
None,
'brief-reminder-on-newton-raphson-s-method'),
('The equations', 2, None, 'the-equations'),
('Simple geometric interpretation',
2,
None,
'simple-geometric-interpretation'),
('Extending to more than one variable',
2,
None,
'extending-to-more-than-one-variable'),
('Steepest descent', 2, None, 'steepest-descent'),
('More on Steepest descent', 2, None, 'more-on-steepest-descent'),
('The ideal', 2, None, 'the-ideal'),
('The sensitiveness of the gradient descent',
2,
None,
'___sec13'),
('Convex functions', 2, None, '___sec14'),
('Convex function', 2, None, '___sec15'),
('Conditions on convex functions', 2, None, '___sec16'),
('More on convex functions', 2, None, '___sec17'),
('Some simple problems', 2, None, '___sec18'),
('Friday September 25', 2, None, '___sec19'),
('Standard steepest descent', 2, None, '___sec20'),
('Gradient method', 2, None, '___sec21'),
('Steepest descent method', 2, None, '___sec22'),
('Steepest descent method', 2, None, '___sec23'),
('Final expressions', 2, None, '___sec24'),
('Steepest descent example', 2, None, '___sec25'),
('Conjugate gradient method', 2, None, '___sec26'),
('Conjugate gradient method', 2, None, '___sec27'),
('Conjugate gradient method', 2, None, '___sec28'),
('Conjugate gradient method', 2, None, '___sec29'),
('Conjugate gradient method and iterations', 2, None, '___sec30'),
('Conjugate gradient method', 2, None, '___sec31'),
('Conjugate gradient method', 2, None, '___sec32'),
('Conjugate gradient method', 2, None, '___sec33'),
('Revisiting our first homework', 2, None, '___sec34'),
('Gradient descent example', 2, None, '___sec35'),
('The derivative of the cost/loss function', 2, None, '___sec36'),
('The Hessian matrix', 2, None, '___sec37'),
('Simple program', 2, None, '___sec38'),
('Gradient Descent Example', 2, None, '___sec39'),
'the-sensitiveness-of-the-gradient-descent'),
('Convex functions', 2, None, 'convex-functions'),
('Convex function', 2, None, 'convex-function'),
('Conditions on convex functions',
2,
None,
'conditions-on-convex-functions'),
('More on convex functions', 2, None, 'more-on-convex-functions'),
('Some simple problems', 2, None, 'some-simple-problems'),
('Friday October 1', 2, None, 'friday-october-1'),
('Standard steepest descent',
2,
None,
'standard-steepest-descent'),
('Gradient method', 2, None, 'gradient-method'),
('Steepest descent method', 2, None, 'steepest-descent-method'),
('Steepest descent method', 2, None, 'steepest-descent-method'),
('Final expressions', 2, None, 'final-expressions'),
('Steepest descent example', 2, None, 'steepest-descent-example'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method and iterations',
2,
None,
'conjugate-gradient-method-and-iterations'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Revisiting our first homework',
2,
None,
'revisiting-our-first-homework'),
('Gradient descent example', 2, None, 'gradient-descent-example'),
('The derivative of the cost/loss function',
2,
None,
'the-derivative-of-the-cost-loss-function'),
('The Hessian matrix', 2, None, 'the-hessian-matrix'),
('Simple program', 2, None, 'simple-program'),
('Gradient Descent Example', 2, None, 'gradient-descent-example'),
('And a corresponding example using _scikit-learn_',
2,
None,
'___sec40'),
('Gradient descent and Ridge', 2, None, '___sec41'),
'and-a-corresponding-example-using-_scikit-learn_'),
('Gradient descent and Ridge',
2,
None,
'gradient-descent-and-ridge'),
('Program example for gradient descent with Ridge Regression',
2,
None,
'___sec42'),
'program-example-for-gradient-descent-with-ridge-regression'),
('Using gradient descent methods, limitations',
2,
None,
'___sec43'),
('Stochastic Gradient Descent', 2, None, '___sec44'),
('Computation of gradients', 2, None, '___sec45'),
('SGD example', 2, None, '___sec46'),
('The gradient step', 2, None, '___sec47'),
('Simple example code', 2, None, '___sec48'),
('When do we stop?', 2, None, '___sec49'),
('Slightly different approach', 2, None, '___sec50'),
('Program for stochastic gradient', 2, None, '___sec51')]}
'using-gradient-descent-methods-limitations'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
('When do we stop?', 2, None, 'when-do-we-stop'),
('Slightly different approach',
2,
None,
'slightly-different-approach'),
('Program for stochastic gradient',
2,
None,
'program-for-stochastic-gradient')]}
end of tocinfo -->
<body>
@@ -146,58 +209,58 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week39-bs001.html#___sec0" style="font-size: 80%;">Plan for week 39</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs002.html#___sec1" style="font-size: 80%;">Thursday September 24</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs003.html#___sec2" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs004.html#___sec3" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs005.html#___sec4" style="font-size: 80%;">The equations to solve</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs006.html#___sec5" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs007.html#___sec6" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs008.html#___sec7" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs009.html#___sec8" style="font-size: 80%;">Simple geometric interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs010.html#___sec9" style="font-size: 80%;">Extending to more than one variable</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs011.html#___sec10" style="font-size: 80%;">Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs012.html#___sec11" style="font-size: 80%;">More on Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs013.html#___sec12" style="font-size: 80%;">The ideal</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs014.html#___sec13" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs015.html#___sec14" style="font-size: 80%;">Convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs016.html#___sec15" style="font-size: 80%;">Convex function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs017.html#___sec16" style="font-size: 80%;">Conditions on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs018.html#___sec17" style="font-size: 80%;">More on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs019.html#___sec18" style="font-size: 80%;">Some simple problems</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs020.html#___sec19" style="font-size: 80%;">Friday September 25</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs021.html#___sec20" style="font-size: 80%;">Standard steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs022.html#___sec21" style="font-size: 80%;">Gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs023.html#___sec22" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#___sec23" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs025.html#___sec24" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs027.html#___sec26" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs028.html#___sec27" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs029.html#___sec28" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs030.html#___sec29" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs031.html#___sec30" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs032.html#___sec31" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#___sec32" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#___sec33" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs035.html#___sec34" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#___sec35" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#___sec36" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs038.html#___sec37" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#___sec38" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#___sec39" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs041.html#___sec40" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#___sec41" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#___sec42" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#___sec43" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#___sec44" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#___sec45" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#___sec46" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs048.html#___sec47" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs049.html#___sec48" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#___sec49" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#___sec50" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs052.html#___sec51" style="font-size: 80%;">Program for stochastic gradient</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs001.html#plan-for-week-39" style="font-size: 80%;">Plan for week 39</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs002.html#thursday-september-30" style="font-size: 80%;">Thursday September 30</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs003.html#optimization-the-central-part-of-any-machine-learning-algortithm" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs004.html#revisiting-our-logistic-regression-case" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs005.html#the-equations-to-solve" style="font-size: 80%;">The equations to solve</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs006.html#solving-using-newton-raphson-s-method" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs007.html#brief-reminder-on-newton-raphson-s-method" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs008.html#the-equations" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs009.html#simple-geometric-interpretation" style="font-size: 80%;">Simple geometric interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs010.html#extending-to-more-than-one-variable" style="font-size: 80%;">Extending to more than one variable</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs011.html#steepest-descent" style="font-size: 80%;">Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs012.html#more-on-steepest-descent" style="font-size: 80%;">More on Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs013.html#the-ideal" style="font-size: 80%;">The ideal</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs014.html#the-sensitiveness-of-the-gradient-descent" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs015.html#convex-functions" style="font-size: 80%;">Convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs016.html#convex-function" style="font-size: 80%;">Convex function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs017.html#conditions-on-convex-functions" style="font-size: 80%;">Conditions on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs018.html#more-on-convex-functions" style="font-size: 80%;">More on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs019.html#some-simple-problems" style="font-size: 80%;">Some simple problems</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs020.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs021.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs022.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs025.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs026.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs031.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs035.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs038.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#simple-program" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs041.html#and-a-corresponding-example-using-_scikit-learn_" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs048.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs049.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs052.html#program-for-stochastic-gradient" style="font-size: 80%;">Program for stochastic gradient</a></li>
</ul>
</li>
@@ -232,7 +295,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 29, 2020</h4></center> <!-- date -->
<center><h4>Sep 25, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -268,13 +331,13 @@ MathJax.Hub.Config({
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
<a href="https://..."><img width="250" align=right src="https://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week39/html/week39-reveal.html
+78 -82
View File
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 39: Optimization and Gradient Methods">
@@ -13,7 +13,7 @@
<!-- reveal.js: http://lab.hakim.se/reveal-js/ -->
<!-- reveal.js: https://lab.hakim.se/reveal-js/ -->
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
@@ -76,10 +76,10 @@ document.getElementsByTagName( 'head' )[0].appendChild( link );
.reveal .alert-block > p, .alert-block > ul {margin-bottom:1em}
/*.reveal .alert li {margin-top: 1em}*/
.reveal .alert-block p+p {margin-top:5px}
/*.reveal .alert-notice { background-image: url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_notice.png); }
.reveal .alert-summary { background-image:url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_summary.png); }
.reveal .alert-warning { background-image: url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_warning.png); }
.reveal .alert-question {background-image:url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_question.png); } */
/*.reveal .alert-notice { background-image: url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_notice.png); }
.reveal .alert-summary { background-image:url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_summary.png); }
.reveal .alert-warning { background-image: url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_warning.png); }
.reveal .alert-question {background-image:url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_question.png); } */
</style>
@@ -148,42 +148,41 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 29, 2020</h4></center> <!-- date -->
<center><h4>Sep 25, 2021</h4></center> <!-- date -->
<br>
<p>
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</section>
<section>
<h2 id="___sec0">Plan for week 39 </h2>
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<p><li> Thursday: Repetition of Logistic regression equations and discussion of Gradient methods</li>
<p><li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
<p><li> Reading recommendations:</li>
</ul>
<p>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 4</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf" target="_blank">Murphy sections 8.3 and 8.5</a>
See <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank">lecture notes for week 39</a>.
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
</section>
<section>
<h2 id="___sec1">Thursday September 24 </h2>
<h2 id="thursday-september-30">Thursday September 30 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage" target="_blank">Overview Video, why do we care about gradient methods?</a>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember24.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember24.pdf" target="_blank">link to handwritten notes</a>.
</section>
<section>
<h2 id="___sec2">Optimization, the central part of any Machine Learning algortithm </h2>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
<p>
Almost every problem in machine learning and data science starts with
@@ -198,7 +197,7 @@ some approximative/numerical method to compute the minimum.
<section>
<h2 id="___sec3">Revisiting our Logistic Regression case </h2>
<h2 id="revisiting-our-logistic-regression-case">Revisiting our Logistic Regression case </h2>
<p>
In our discussion on Logistic Regression we studied the
@@ -222,7 +221,7 @@ where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in
<section>
<h2 id="___sec4">The equations to solve </h2>
<h2 id="the-equations-to-solve">The equations to solve </h2>
<p>
Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \)
@@ -252,7 +251,7 @@ This defines what is called the Hessian matrix.
<section>
<h2 id="___sec5">Solving using Newton-Raphson's method </h2>
<h2 id="solving-using-newton-raphson-s-method">Solving using Newton-Raphson's method </h2>
<p>
If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.
@@ -282,7 +281,7 @@ If we can compute these matrices, in particular the Hessian, the above is often
<section>
<h2 id="___sec6">Brief reminder on Newton-Raphson's method </h2>
<h2 id="brief-reminder-on-newton-raphson-s-method">Brief reminder on Newton-Raphson's method </h2>
<p>
Let us quickly remind ourselves how we derive the above method.
@@ -299,7 +298,7 @@ normally discourage the use of this method.
<section>
<h2 id="___sec7">The equations </h2>
<h2 id="the-equations">The equations </h2>
<p>
The Newton-Raphson formula consists geometrically of extending the
@@ -343,7 +342,7 @@ $$
<section>
<h2 id="___sec8">Simple geometric interpretation </h2>
<h2 id="simple-geometric-interpretation">Simple geometric interpretation </h2>
<p>
The above is Newton-Raphson's method. It has a simple geometric
@@ -361,7 +360,7 @@ vanishes, then Newton-Raphson may fail totally
<section>
<h2 id="___sec9">Extending to more than one variable </h2>
<h2 id="extending-to-more-than-one-variable">Extending to more than one variable </h2>
<p>
Newton's method can be generalized to systems of several non-linear equations
@@ -426,7 +425,7 @@ more than two non-linear equations. In our case, the Jacobian matrix is given by
<section>
<h2 id="___sec10">Steepest descent </h2>
<h2 id="steepest-descent">Steepest descent </h2>
<p>
The basic idea of gradient descent is
@@ -452,7 +451,7 @@ we are always moving towards smaller function values, i.e a minimum.
<section>
<h2 id="___sec11">More on Steepest descent </h2>
<h2 id="more-on-steepest-descent">More on Steepest descent </h2>
<p>
The previous observation is the basis of the method of steepest
@@ -473,7 +472,7 @@ the learning rate within the context of Machine Learning.
<section>
<h2 id="___sec12">The ideal </h2>
<h2 id="the-ideal">The ideal </h2>
<p>
Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global
@@ -499,7 +498,7 @@ Note that the gradient is a function of \( \mathbf{x} =
<section>
<h2 id="___sec13">The sensitiveness of the gradient descent </h2>
<h2 id="the-sensitiveness-of-the-gradient-descent">The sensitiveness of the gradient descent </h2>
<p>
The gradient descent method
@@ -518,7 +517,7 @@ randomness. One such method is that of Stochastic Gradient Descent
<section>
<h2 id="___sec14">Convex functions </h2>
<h2 id="convex-functions">Convex functions </h2>
<p>
Ideally we want our cost/loss function to be convex(concave).
@@ -538,7 +537,7 @@ regular polygons (triangles, rectangles, pentagons, etc...).
<section>
<h2 id="___sec15">Convex function </h2>
<h2 id="convex-function">Convex function </h2>
<p>
<b>Convex function</b>: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if <p>&nbsp;<br>
@@ -548,7 +547,7 @@ $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$
<section>
<h2 id="___sec16">Conditions on convex functions </h2>
<h2 id="conditions-on-convex-functions">Conditions on convex functions </h2>
<p>
In the following we state first and second-order conditions which
@@ -558,7 +557,7 @@ details and proofs we refer to: <a href="http://stanford.edu/boyd/cvxbook/, 2004
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>First order condition.</b>
<b>First order condition</b>
<p>
Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
@@ -574,7 +573,7 @@ note that it is always below the graph.
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Second order condition.</b>
<b>Second order condition</b>
<p>
Assume that \( f \) is twice
differentiable, i.e the Hessian matrix exists at each point in
@@ -590,7 +589,7 @@ This condition is particularly useful since it gives us an procedure for determi
<section>
<h2 id="___sec17">More on convex functions </h2>
<h2 id="more-on-convex-functions">More on convex functions </h2>
<p>
The next result is of great importance to us and the reason why we are
@@ -606,7 +605,7 @@ is convex the following result provides invaluable information:
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Any minimum is global for convex functions.</b>
<b>Any minimum is global for convex functions</b>
<p>
Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
is minimal, where \( f \) is convex and differentiable. Then, any point
@@ -619,7 +618,7 @@ This result means that if we know that the cost/loss function is convex and we a
<section>
<h2 id="___sec18">Some simple problems </h2>
<h2 id="some-simple-problems">Some simple problems </h2>
<ol>
<p><li> Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.</li>
@@ -646,15 +645,12 @@ Using the definition of convexity, try to show that a function satisfying the pr
<section>
<h2 id="___sec19">Friday September 25 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember25.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember25.pdf" target="_blank">link to handwritten notes</a>.
<h2 id="friday-october-1">Friday October 1 </h2>
</section>
<section>
<h2 id="___sec20">Standard steepest descent </h2>
<h2 id="standard-steepest-descent">Standard steepest descent </h2>
<p>
Before we proceed, we would like to discuss the approach called the
@@ -694,7 +690,7 @@ When we have found the exact solution, \( \boldsymbol{r}=0 \).
<section>
<h2 id="___sec21">Gradient method </h2>
<h2 id="gradient-method">Gradient method </h2>
<p>
The residual is zero when we reach the minimum of the quadratic equation
@@ -713,7 +709,7 @@ symmetric. This defines also the Hessian and we want it to be positive definit
<section>
<h2 id="___sec22">Steepest descent method </h2>
<h2 id="steepest-descent-method">Steepest descent method </h2>
<p>
We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \).
@@ -740,7 +736,7 @@ instead.
<section>
<h2 id="___sec23">Steepest descent method </h2>
<h2 id="steepest-descent-method">Steepest descent method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -773,7 +769,7 @@ and
<section>
<h2 id="___sec24">Final expressions </h2>
<h2 id="final-expressions">Final expressions </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -825,12 +821,12 @@ $$
<section>
<h2 id="___sec25">Steepest descent example </h2>
<h2 id="steepest-descent-example">Steepest descent example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy.linalg</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">la</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">scipy.optimize</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sopt</span>
@@ -856,7 +852,7 @@ And then as countor plot
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>pt.axis(<span style="color: #CD5555">&quot;equal&quot;</span>)
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>pt.axis(<span style="color: #CD5555">&quot;equal&quot;</span>)
pt.contour(xmesh, ymesh, fmesh)
guesses = [np.array([<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2.</span>/<span style="color: #B452CD">5</span>])]
</pre></div>
@@ -865,7 +861,7 @@ Find guesses
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>x = guesses[-<span style="color: #B452CD">1</span>]
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>x = guesses[-<span style="color: #B452CD">1</span>]
s = -df(x)
</pre></div>
<p>
@@ -873,7 +869,7 @@ Run it!
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1d</span>(alpha):
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1d</span>(alpha):
<span style="color: #8B008B; font-weight: bold">return</span> f(x + alpha*s)
alpha_opt = sopt.golden(f1d)
@@ -886,7 +882,7 @@ What happened?
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>pt.axis(<span style="color: #CD5555">&quot;equal&quot;</span>)
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>pt.axis(<span style="color: #CD5555">&quot;equal&quot;</span>)
pt.contour(xmesh, ymesh, fmesh, <span style="color: #B452CD">50</span>)
it_array = np.array(guesses)
pt.plot(it_array.T[<span style="color: #B452CD">0</span>], it_array.T[<span style="color: #B452CD">1</span>], <span style="color: #CD5555">&quot;x-&quot;</span>)
@@ -895,7 +891,7 @@ pt.plot(it_array.T[<span style="color: #B452CD">0</span>], it_array.T[<span styl
<section>
<h2 id="___sec26">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -928,7 +924,7 @@ this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s
<section>
<h2 id="___sec27">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -947,7 +943,7 @@ which is zero unless \( i=j \).
<section>
<h2 id="___sec28">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -977,7 +973,7 @@ $$
<section>
<h2 id="___sec29">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1014,7 +1010,7 @@ $$
<section>
<h2 id="___sec30">Conjugate gradient method and iterations </h2>
<h2 id="conjugate-gradient-method-and-iterations">Conjugate gradient method and iterations </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1051,7 +1047,7 @@ instead.
<section>
<h2 id="___sec31">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1084,7 +1080,7 @@ hence the name conjugate gradient method.
<section>
<h2 id="___sec32">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1116,7 +1112,7 @@ $$
<section>
<h2 id="___sec33">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1161,7 +1157,7 @@ $$
<section>
<h2 id="___sec34">Revisiting our first homework </h2>
<h2 id="revisiting-our-first-homework">Revisiting our first homework </h2>
<p>
We will use linear regression as a case study for the gradient descent
@@ -1181,7 +1177,7 @@ We revisit an example similar to what we had in the first homework set. We had a
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #B452CD">1</span>)
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(m,<span style="color: #B452CD">1</span>)
</pre></div>
<p>
@@ -1203,7 +1199,7 @@ $$
<section>
<h2 id="___sec35">Gradient descent example </h2>
<h2 id="gradient-descent-example">Gradient descent example </h2>
<p>
Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
@@ -1232,7 +1228,7 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
<section>
<h2 id="___sec36">The derivative of the cost/loss function </h2>
<h2 id="the-derivative-of-the-cost-loss-function">The derivative of the cost/loss function </h2>
<p>
Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as
@@ -1249,7 +1245,7 @@ where \( X \) is the design matrix defined above.
<section>
<h2 id="___sec37">The Hessian matrix </h2>
<h2 id="the-hessian-matrix">The Hessian matrix </h2>
The Hessian matrix of \( C(\beta) \) is given by
<p>&nbsp;<br>
$$
@@ -1265,7 +1261,7 @@ This result implies that \( C(\beta) \) is a convex function since the matrix \(
<section>
<h2 id="___sec38">Simple program </h2>
<h2 id="simple-program">Simple program </h2>
<p>
We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to
@@ -1287,14 +1283,14 @@ And finally we can compare our solution for \( \beta \) with the analytic result
<section>
<h2 id="___sec39">Gradient Descent Example </h2>
<h2 id="gradient-descent-example">Gradient Descent Example </h2>
<p>
Here our simple example
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
@@ -1344,12 +1340,12 @@ plt.show()
<section>
<h2 id="___sec40">And a corresponding example using <b>scikit-learn</b> </h2>
<h2 id="and-a-corresponding-example-using-_scikit-learn_">And a corresponding example using <b>scikit-learn</b> </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
@@ -1370,7 +1366,7 @@ sgdreg.fit(x,y.ravel())
<section>
<h2 id="___sec41">Gradient descent and Ridge </h2>
<h2 id="gradient-descent-and-ridge">Gradient descent and Ridge </h2>
<p>
We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \beta \),
@@ -1401,11 +1397,11 @@ $$
<section>
<h2 id="___sec42">Program example for gradient descent with Ridge Regression </h2>
<h2 id="program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">mpl_toolkits.mplot3d</span> <span style="color: #8B008B; font-weight: bold">import</span> Axes3D
@@ -1453,7 +1449,7 @@ plt.show()
<section>
<h2 id="___sec43">Using gradient descent methods, limitations </h2>
<h2 id="using-gradient-descent-methods-limitations">Using gradient descent methods, limitations </h2>
<ul>
<p><li> <b>Gradient descent (GD) finds local minima of our function</b>. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our cost/loss/risk function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.</li>
@@ -1467,7 +1463,7 @@ plt.show()
<section>
<h2 id="___sec44">Stochastic Gradient Descent </h2>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -1487,7 +1483,7 @@ $$
<section>
<h2 id="___sec45">Computation of gradients </h2>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>
This in turn means that the gradient can be
@@ -1509,7 +1505,7 @@ minibatches. We denote these minibatches by \( B_k \) where
<section>
<h2 id="___sec46">SGD example </h2>
<h2 id="sgd-example">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -1535,7 +1531,7 @@ $$
<section>
<h2 id="___sec47">The gradient step </h2>
<h2 id="the-gradient-step">The gradient step </h2>
<p>
Thus a gradient descent step now looks like
@@ -1556,12 +1552,12 @@ the number of minibatches, as exemplified in the code below.
<section>
<h2 id="___sec48">Simple example code </h2>
<h2 id="simple-example-code">Simple example code </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
n = <span style="color: #B452CD">100</span> <span style="color: #228B22">#100 datapoints </span>
M = <span style="color: #B452CD">5</span> <span style="color: #228B22">#size of each minibatch</span>
@@ -1588,7 +1584,7 @@ all \( n \) datapoints.
<section>
<h2 id="___sec49">When do we stop? </h2>
<h2 id="when-do-we-stop">When do we stop? </h2>
<p>
A natural question is when do we stop the search for a new minimum?
@@ -1605,7 +1601,7 @@ gave the lowest value.
<section>
<h2 id="___sec50">Slightly different approach </h2>
<h2 id="slightly-different-approach">Slightly different approach </h2>
<p>
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -1627,7 +1623,7 @@ function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">step_length</span>(t,t0,t1):
<span style="color: #8B008B; font-weight: bold">return</span> t0/(t+t1)
@@ -1656,12 +1652,12 @@ j = <span style="color: #B452CD">0</span>
<section>
<h2 id="___sec51">Program for stochastic gradient </h2>
<h2 id="program-for-stochastic-gradient">Program for stochastic gradient </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">math</span> <span style="color: #8B008B; font-weight: bold">import</span> exp, sqrt
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
doc/pub/week39/html/week39-solarized.html
+195 -136
View File
@@ -1,19 +1,19 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 39: Optimization and Gradient Methods">
<title>Week 39: Optimization and Gradient Methods</title>
<link href="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<link href="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<link href="https://thomasf.github.io/solarized-css/solarized-light.min.css" rel="stylesheet">
@@ -48,10 +48,10 @@ pre {
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_question.png); }
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
@@ -61,74 +61,137 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 39', 2, None, '___sec0'),
('Thursday September 24', 2, None, '___sec1'),
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
('Thursday September 30', 2, None, 'thursday-september-30'),
('Optimization, the central part of any Machine Learning '
'algortithm',
2,
None,
'___sec2'),
('Revisiting our Logistic Regression case', 2, None, '___sec3'),
('The equations to solve', 2, None, '___sec4'),
("Solving using Newton-Raphson's method", 2, None, '___sec5'),
("Brief reminder on Newton-Raphson's method", 2, None, '___sec6'),
('The equations', 2, None, '___sec7'),
('Simple geometric interpretation', 2, None, '___sec8'),
('Extending to more than one variable', 2, None, '___sec9'),
('Steepest descent', 2, None, '___sec10'),
('More on Steepest descent', 2, None, '___sec11'),
('The ideal', 2, None, '___sec12'),
'optimization-the-central-part-of-any-machine-learning-algortithm'),
('Revisiting our Logistic Regression case',
2,
None,
'revisiting-our-logistic-regression-case'),
('The equations to solve', 2, None, 'the-equations-to-solve'),
("Solving using Newton-Raphson's method",
2,
None,
'solving-using-newton-raphson-s-method'),
("Brief reminder on Newton-Raphson's method",
2,
None,
'brief-reminder-on-newton-raphson-s-method'),
('The equations', 2, None, 'the-equations'),
('Simple geometric interpretation',
2,
None,
'simple-geometric-interpretation'),
('Extending to more than one variable',
2,
None,
'extending-to-more-than-one-variable'),
('Steepest descent', 2, None, 'steepest-descent'),
('More on Steepest descent', 2, None, 'more-on-steepest-descent'),
('The ideal', 2, None, 'the-ideal'),
('The sensitiveness of the gradient descent',
2,
None,
'___sec13'),
('Convex functions', 2, None, '___sec14'),
('Convex function', 2, None, '___sec15'),
('Conditions on convex functions', 2, None, '___sec16'),
('More on convex functions', 2, None, '___sec17'),
('Some simple problems', 2, None, '___sec18'),
('Friday September 25', 2, None, '___sec19'),
('Standard steepest descent', 2, None, '___sec20'),
('Gradient method', 2, None, '___sec21'),
('Steepest descent method', 2, None, '___sec22'),
('Steepest descent method', 2, None, '___sec23'),
('Final expressions', 2, None, '___sec24'),
('Steepest descent example', 2, None, '___sec25'),
('Conjugate gradient method', 2, None, '___sec26'),
('Conjugate gradient method', 2, None, '___sec27'),
('Conjugate gradient method', 2, None, '___sec28'),
('Conjugate gradient method', 2, None, '___sec29'),
('Conjugate gradient method and iterations', 2, None, '___sec30'),
('Conjugate gradient method', 2, None, '___sec31'),
('Conjugate gradient method', 2, None, '___sec32'),
('Conjugate gradient method', 2, None, '___sec33'),
('Revisiting our first homework', 2, None, '___sec34'),
('Gradient descent example', 2, None, '___sec35'),
('The derivative of the cost/loss function', 2, None, '___sec36'),
('The Hessian matrix', 2, None, '___sec37'),
('Simple program', 2, None, '___sec38'),
('Gradient Descent Example', 2, None, '___sec39'),
'the-sensitiveness-of-the-gradient-descent'),
('Convex functions', 2, None, 'convex-functions'),
('Convex function', 2, None, 'convex-function'),
('Conditions on convex functions',
2,
None,
'conditions-on-convex-functions'),
('More on convex functions', 2, None, 'more-on-convex-functions'),
('Some simple problems', 2, None, 'some-simple-problems'),
('Friday October 1', 2, None, 'friday-october-1'),
('Standard steepest descent',
2,
None,
'standard-steepest-descent'),
('Gradient method', 2, None, 'gradient-method'),
('Steepest descent method', 2, None, 'steepest-descent-method'),
('Steepest descent method', 2, None, 'steepest-descent-method'),
('Final expressions', 2, None, 'final-expressions'),
('Steepest descent example', 2, None, 'steepest-descent-example'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method and iterations',
2,
None,
'conjugate-gradient-method-and-iterations'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Revisiting our first homework',
2,
None,
'revisiting-our-first-homework'),
('Gradient descent example', 2, None, 'gradient-descent-example'),
('The derivative of the cost/loss function',
2,
None,
'the-derivative-of-the-cost-loss-function'),
('The Hessian matrix', 2, None, 'the-hessian-matrix'),
('Simple program', 2, None, 'simple-program'),
('Gradient Descent Example', 2, None, 'gradient-descent-example'),
('And a corresponding example using _scikit-learn_',
2,
None,
'___sec40'),
('Gradient descent and Ridge', 2, None, '___sec41'),
'and-a-corresponding-example-using-_scikit-learn_'),
('Gradient descent and Ridge',
2,
None,
'gradient-descent-and-ridge'),
('Program example for gradient descent with Ridge Regression',
2,
None,
'___sec42'),
'program-example-for-gradient-descent-with-ridge-regression'),
('Using gradient descent methods, limitations',
2,
None,
'___sec43'),
('Stochastic Gradient Descent', 2, None, '___sec44'),
('Computation of gradients', 2, None, '___sec45'),
('SGD example', 2, None, '___sec46'),
('The gradient step', 2, None, '___sec47'),
('Simple example code', 2, None, '___sec48'),
('When do we stop?', 2, None, '___sec49'),
('Slightly different approach', 2, None, '___sec50'),
('Program for stochastic gradient', 2, None, '___sec51')]}
'using-gradient-descent-methods-limitations'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
('When do we stop?', 2, None, 'when-do-we-stop'),
('Slightly different approach',
2,
None,
'slightly-different-approach'),
('Program for stochastic gradient',
2,
None,
'program-for-stochastic-gradient')]}
end of tocinfo -->
<body>
@@ -170,35 +233,34 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 29, 2020</h4></center> <!-- date -->
<center><h4>Sep 25, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Plan for week 39 </h2>
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<li> Thursday: Repetition of Logistic regression equations and discussion of Gradient methods</li>
<li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
<li> Reading recommendations:</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 4</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf" target="_blank">Murphy sections 8.3 and 8.5</a>
See <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank">lecture notes for week 39</a>.
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Thursday September 24 </h2>
<h2 id="thursday-september-30">Thursday September 30 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage" target="_blank">Overview Video, why do we care about gradient methods?</a>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember24.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember24.pdf" target="_blank">link to handwritten notes</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Optimization, the central part of any Machine Learning algortithm </h2>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
<p>
Almost every problem in machine learning and data science starts with
@@ -213,7 +275,7 @@ some approximative/numerical method to compute the minimum.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Revisiting our Logistic Regression case </h2>
<h2 id="revisiting-our-logistic-regression-case">Revisiting our Logistic Regression case </h2>
<p>
In our discussion on Logistic Regression we studied the
@@ -235,7 +297,7 @@ where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">The equations to solve </h2>
<h2 id="the-equations-to-solve">The equations to solve </h2>
<p>
Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \)
@@ -261,7 +323,7 @@ This defines what is called the Hessian matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Solving using Newton-Raphson's method </h2>
<h2 id="solving-using-newton-raphson-s-method">Solving using Newton-Raphson's method </h2>
<p>
If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.
@@ -287,7 +349,7 @@ If we can compute these matrices, in particular the Hessian, the above is often
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Brief reminder on Newton-Raphson's method </h2>
<h2 id="brief-reminder-on-newton-raphson-s-method">Brief reminder on Newton-Raphson's method </h2>
<p>
Let us quickly remind ourselves how we derive the above method.
@@ -304,7 +366,7 @@ normally discourage the use of this method.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">The equations </h2>
<h2 id="the-equations">The equations </h2>
<p>
The Newton-Raphson formula consists geometrically of extending the
@@ -340,7 +402,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Simple geometric interpretation </h2>
<h2 id="simple-geometric-interpretation">Simple geometric interpretation </h2>
<p>
The above is Newton-Raphson's method. It has a simple geometric
@@ -358,7 +420,7 @@ vanishes, then Newton-Raphson may fail totally
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Extending to more than one variable </h2>
<h2 id="extending-to-more-than-one-variable">Extending to more than one variable </h2>
<p>
Newton's method can be generalized to systems of several non-linear equations
@@ -413,7 +475,7 @@ more than two non-linear equations. In our case, the Jacobian matrix is given by
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Steepest descent </h2>
<h2 id="steepest-descent">Steepest descent </h2>
<p>
The basic idea of gradient descent is
@@ -437,7 +499,7 @@ we are always moving towards smaller function values, i.e a minimum.
<p>
<!-- !split -->
<h2 id="___sec11">More on Steepest descent </h2>
<h2 id="more-on-steepest-descent">More on Steepest descent </h2>
<p>
The previous observation is the basis of the method of steepest
@@ -456,7 +518,7 @@ the learning rate within the context of Machine Learning.
<p>
<!-- !split -->
<h2 id="___sec12">The ideal </h2>
<h2 id="the-ideal">The ideal </h2>
<p>
Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global
@@ -482,7 +544,7 @@ Note that the gradient is a function of \( \mathbf{x} =
<p>
<!-- !split -->
<h2 id="___sec13">The sensitiveness of the gradient descent </h2>
<h2 id="the-sensitiveness-of-the-gradient-descent">The sensitiveness of the gradient descent </h2>
<p>
The gradient descent method
@@ -501,7 +563,7 @@ randomness. One such method is that of Stochastic Gradient Descent
<p>
<!-- !split -->
<h2 id="___sec14">Convex functions </h2>
<h2 id="convex-functions">Convex functions </h2>
<p>
Ideally we want our cost/loss function to be convex(concave).
@@ -521,7 +583,7 @@ regular polygons (triangles, rectangles, pentagons, etc...).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Convex function </h2>
<h2 id="convex-function">Convex function </h2>
<p>
<b>Convex function</b>: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.
@@ -529,7 +591,7 @@ regular polygons (triangles, rectangles, pentagons, etc...).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Conditions on convex functions </h2>
<h2 id="conditions-on-convex-functions">Conditions on convex functions </h2>
<p>
In the following we state first and second-order conditions which
@@ -539,7 +601,7 @@ details and proofs we refer to: <a href="http://stanford.edu/boyd/cvxbook/, 2004
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>First order condition.</b>
<b>First order condition</b>
<p>
Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
@@ -554,7 +616,7 @@ note that it is always below the graph.
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Second order condition.</b>
<b>Second order condition</b>
<p>
Assume that \( f \) is twice
differentiable, i.e the Hessian matrix exists at each point in
@@ -571,7 +633,7 @@ This condition is particularly useful since it gives us an procedure for determi
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">More on convex functions </h2>
<h2 id="more-on-convex-functions">More on convex functions </h2>
<p>
The next result is of great importance to us and the reason why we are
@@ -587,7 +649,7 @@ is convex the following result provides invaluable information:
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Any minimum is global for convex functions.</b>
<b>Any minimum is global for convex functions</b>
<p>
Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
is minimal, where \( f \) is convex and differentiable. Then, any point
@@ -601,7 +663,7 @@ This result means that if we know that the cost/loss function is convex and we a
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Some simple problems </h2>
<h2 id="some-simple-problems">Some simple problems </h2>
<ol>
<li> Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.</li>
@@ -628,15 +690,12 @@ Using the definition of convexity, try to show that a function satisfying the pr
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Friday September 25 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember25.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember25.pdf" target="_blank">link to handwritten notes</a>.
<h2 id="friday-october-1">Friday October 1 </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">Standard steepest descent </h2>
<h2 id="standard-steepest-descent">Standard steepest descent </h2>
<p>
Before we proceed, we would like to discuss the approach called the
@@ -672,7 +731,7 @@ When we have found the exact solution, \( \boldsymbol{r}=0 \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec21">Gradient method </h2>
<h2 id="gradient-method">Gradient method </h2>
<p>
The residual is zero when we reach the minimum of the quadratic equation
@@ -689,7 +748,7 @@ symmetric. This defines also the Hessian and we want it to be positive definit
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">Steepest descent method </h2>
<h2 id="steepest-descent-method">Steepest descent method </h2>
<p>
We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \).
@@ -712,7 +771,7 @@ instead.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Steepest descent method </h2>
<h2 id="steepest-descent-method">Steepest descent method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -742,7 +801,7 @@ and
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Final expressions </h2>
<h2 id="final-expressions">Final expressions </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -785,12 +844,12 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Steepest descent example </h2>
<h2 id="steepest-descent-example">Steepest descent example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy.linalg</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">la</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">scipy.optimize</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sopt</span>
@@ -816,7 +875,7 @@ And then as countor plot
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>pt.axis(<span style="color: #CD5555">&quot;equal&quot;</span>)
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>pt.axis(<span style="color: #CD5555">&quot;equal&quot;</span>)
pt.contour(xmesh, ymesh, fmesh)
guesses = [np.array([<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2.</span>/<span style="color: #B452CD">5</span>])]
</pre></div>
@@ -825,7 +884,7 @@ Find guesses
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>x = guesses[-<span style="color: #B452CD">1</span>]
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>x = guesses[-<span style="color: #B452CD">1</span>]
s = -df(x)
</pre></div>
<p>
@@ -833,7 +892,7 @@ Run it!
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1d</span>(alpha):
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">f1d</span>(alpha):
<span style="color: #8B008B; font-weight: bold">return</span> f(x + alpha*s)
alpha_opt = sopt.golden(f1d)
@@ -846,7 +905,7 @@ What happened?
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>pt.axis(<span style="color: #CD5555">&quot;equal&quot;</span>)
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>pt.axis(<span style="color: #CD5555">&quot;equal&quot;</span>)
pt.contour(xmesh, ymesh, fmesh, <span style="color: #B452CD">50</span>)
it_array = np.array(guesses)
pt.plot(it_array.T[<span style="color: #B452CD">0</span>], it_array.T[<span style="color: #B452CD">1</span>], <span style="color: #CD5555">&quot;x-&quot;</span>)
@@ -854,7 +913,7 @@ pt.plot(it_array.T[<span style="color: #B452CD">0</span>], it_array.T[<span styl
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -884,7 +943,7 @@ this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -902,7 +961,7 @@ which is zero unless \( i=j \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -929,7 +988,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -961,7 +1020,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Conjugate gradient method and iterations </h2>
<h2 id="conjugate-gradient-method-and-iterations">Conjugate gradient method and iterations </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -997,7 +1056,7 @@ instead.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1027,7 +1086,7 @@ hence the name conjugate gradient method.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec32">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1056,7 +1115,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec33">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1094,7 +1153,7 @@ $$
<p>
<!-- !split -->
<h2 id="___sec34">Revisiting our first homework </h2>
<h2 id="revisiting-our-first-homework">Revisiting our first homework </h2>
<p>
We will use linear regression as a case study for the gradient descent
@@ -1113,7 +1172,7 @@ We revisit an example similar to what we had in the first homework set. We had a
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #B452CD">1</span>)
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(m,<span style="color: #B452CD">1</span>)
</pre></div>
<p>
@@ -1131,7 +1190,7 @@ $$
<p>
<!-- !split -->
<h2 id="___sec35">Gradient descent example </h2>
<h2 id="gradient-descent-example">Gradient descent example </h2>
<p>
Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
@@ -1156,7 +1215,7 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">The derivative of the cost/loss function </h2>
<h2 id="the-derivative-of-the-cost-loss-function">The derivative of the cost/loss function </h2>
<p>
Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as
@@ -1171,7 +1230,7 @@ where \( X \) is the design matrix defined above.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">The Hessian matrix </h2>
<h2 id="the-hessian-matrix">The Hessian matrix </h2>
The Hessian matrix of \( C(\beta) \) is given by
$$
\boldsymbol{H} \equiv \begin{bmatrix}
@@ -1185,7 +1244,7 @@ This result implies that \( C(\beta) \) is a convex function since the matrix \(
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec38">Simple program </h2>
<h2 id="simple-program">Simple program </h2>
<p>
We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to
@@ -1205,14 +1264,14 @@ And finally we can compare our solution for \( \beta \) with the analytic result
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Gradient Descent Example </h2>
<h2 id="gradient-descent-example">Gradient Descent Example </h2>
<p>
Here our simple example
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
@@ -1261,12 +1320,12 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">And a corresponding example using <b>scikit-learn</b> </h2>
<h2 id="and-a-corresponding-example-using-_scikit-learn_">And a corresponding example using <b>scikit-learn</b> </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
@@ -1286,7 +1345,7 @@ sgdreg.fit(x,y.ravel())
<p>
<!-- !split -->
<h2 id="___sec41">Gradient descent and Ridge </h2>
<h2 id="gradient-descent-and-ridge">Gradient descent and Ridge </h2>
<p>
We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \beta \),
@@ -1311,11 +1370,11 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec42">Program example for gradient descent with Ridge Regression </h2>
<h2 id="program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">mpl_toolkits.mplot3d</span> <span style="color: #8B008B; font-weight: bold">import</span> Axes3D
@@ -1362,7 +1421,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec43">Using gradient descent methods, limitations </h2>
<h2 id="using-gradient-descent-methods-limitations">Using gradient descent methods, limitations </h2>
<ul>
<li> <b>Gradient descent (GD) finds local minima of our function</b>. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our cost/loss/risk function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.</li>
@@ -1375,7 +1434,7 @@ plt.show()
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec44">Stochastic Gradient Descent </h2>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -1393,7 +1452,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec45">Computation of gradients </h2>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>
This in turn means that the gradient can be
@@ -1413,7 +1472,7 @@ minibatches. We denote these minibatches by \( B_k \) where
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec46">SGD example </h2>
<h2 id="sgd-example">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -1437,7 +1496,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec47">The gradient step </h2>
<h2 id="the-gradient-step">The gradient step </h2>
<p>
Thus a gradient descent step now looks like
@@ -1456,12 +1515,12 @@ the number of minibatches, as exemplified in the code below.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">Simple example code </h2>
<h2 id="simple-example-code">Simple example code </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
n = <span style="color: #B452CD">100</span> <span style="color: #228B22">#100 datapoints </span>
M = <span style="color: #B452CD">5</span> <span style="color: #228B22">#size of each minibatch</span>
@@ -1488,7 +1547,7 @@ all \( n \) datapoints.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">When do we stop? </h2>
<h2 id="when-do-we-stop">When do we stop? </h2>
<p>
A natural question is when do we stop the search for a new minimum?
@@ -1505,7 +1564,7 @@ gave the lowest value.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">Slightly different approach </h2>
<h2 id="slightly-different-approach">Slightly different approach </h2>
<p>
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -1525,7 +1584,7 @@ function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">step_length</span>(t,t0,t1):
<span style="color: #8B008B; font-weight: bold">return</span> t0/(t+t1)
@@ -1553,12 +1612,12 @@ j = <span style="color: #B452CD">0</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Program for stochastic gradient </h2>
<h2 id="program-for-stochastic-gradient">Program for stochastic gradient </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">math</span> <span style="color: #8B008B; font-weight: bold">import</span> exp, sqrt
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">random</span> <span style="color: #8B008B; font-weight: bold">import</span> random, seed
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
@@ -1632,7 +1691,7 @@ plt.show()
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week39/html/week39.html
+193 -134
View File
@@ -1,11 +1,11 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 39: Optimization and Gradient Methods">
@@ -53,10 +53,10 @@ hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_question.png); }
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
@@ -66,74 +66,137 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 39', 2, None, '___sec0'),
('Thursday September 24', 2, None, '___sec1'),
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
('Thursday September 30', 2, None, 'thursday-september-30'),
('Optimization, the central part of any Machine Learning '
'algortithm',
2,
None,
'___sec2'),
('Revisiting our Logistic Regression case', 2, None, '___sec3'),
('The equations to solve', 2, None, '___sec4'),
("Solving using Newton-Raphson's method", 2, None, '___sec5'),
("Brief reminder on Newton-Raphson's method", 2, None, '___sec6'),
('The equations', 2, None, '___sec7'),
('Simple geometric interpretation', 2, None, '___sec8'),
('Extending to more than one variable', 2, None, '___sec9'),
('Steepest descent', 2, None, '___sec10'),
('More on Steepest descent', 2, None, '___sec11'),
('The ideal', 2, None, '___sec12'),
'optimization-the-central-part-of-any-machine-learning-algortithm'),
('Revisiting our Logistic Regression case',
2,
None,
'revisiting-our-logistic-regression-case'),
('The equations to solve', 2, None, 'the-equations-to-solve'),
("Solving using Newton-Raphson's method",
2,
None,
'solving-using-newton-raphson-s-method'),
("Brief reminder on Newton-Raphson's method",
2,
None,
'brief-reminder-on-newton-raphson-s-method'),
('The equations', 2, None, 'the-equations'),
('Simple geometric interpretation',
2,
None,
'simple-geometric-interpretation'),
('Extending to more than one variable',
2,
None,
'extending-to-more-than-one-variable'),
('Steepest descent', 2, None, 'steepest-descent'),
('More on Steepest descent', 2, None, 'more-on-steepest-descent'),
('The ideal', 2, None, 'the-ideal'),
('The sensitiveness of the gradient descent',
2,
None,
'___sec13'),
('Convex functions', 2, None, '___sec14'),
('Convex function', 2, None, '___sec15'),
('Conditions on convex functions', 2, None, '___sec16'),
('More on convex functions', 2, None, '___sec17'),
('Some simple problems', 2, None, '___sec18'),
('Friday September 25', 2, None, '___sec19'),
('Standard steepest descent', 2, None, '___sec20'),
('Gradient method', 2, None, '___sec21'),
('Steepest descent method', 2, None, '___sec22'),
('Steepest descent method', 2, None, '___sec23'),
('Final expressions', 2, None, '___sec24'),
('Steepest descent example', 2, None, '___sec25'),
('Conjugate gradient method', 2, None, '___sec26'),
('Conjugate gradient method', 2, None, '___sec27'),
('Conjugate gradient method', 2, None, '___sec28'),
('Conjugate gradient method', 2, None, '___sec29'),
('Conjugate gradient method and iterations', 2, None, '___sec30'),
('Conjugate gradient method', 2, None, '___sec31'),
('Conjugate gradient method', 2, None, '___sec32'),
('Conjugate gradient method', 2, None, '___sec33'),
('Revisiting our first homework', 2, None, '___sec34'),
('Gradient descent example', 2, None, '___sec35'),
('The derivative of the cost/loss function', 2, None, '___sec36'),
('The Hessian matrix', 2, None, '___sec37'),
('Simple program', 2, None, '___sec38'),
('Gradient Descent Example', 2, None, '___sec39'),
'the-sensitiveness-of-the-gradient-descent'),
('Convex functions', 2, None, 'convex-functions'),
('Convex function', 2, None, 'convex-function'),
('Conditions on convex functions',
2,
None,
'conditions-on-convex-functions'),
('More on convex functions', 2, None, 'more-on-convex-functions'),
('Some simple problems', 2, None, 'some-simple-problems'),
('Friday October 1', 2, None, 'friday-october-1'),
('Standard steepest descent',
2,
None,
'standard-steepest-descent'),
('Gradient method', 2, None, 'gradient-method'),
('Steepest descent method', 2, None, 'steepest-descent-method'),
('Steepest descent method', 2, None, 'steepest-descent-method'),
('Final expressions', 2, None, 'final-expressions'),
('Steepest descent example', 2, None, 'steepest-descent-example'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method and iterations',
2,
None,
'conjugate-gradient-method-and-iterations'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Conjugate gradient method',
2,
None,
'conjugate-gradient-method'),
('Revisiting our first homework',
2,
None,
'revisiting-our-first-homework'),
('Gradient descent example', 2, None, 'gradient-descent-example'),
('The derivative of the cost/loss function',
2,
None,
'the-derivative-of-the-cost-loss-function'),
('The Hessian matrix', 2, None, 'the-hessian-matrix'),
('Simple program', 2, None, 'simple-program'),
('Gradient Descent Example', 2, None, 'gradient-descent-example'),
('And a corresponding example using _scikit-learn_',
2,
None,
'___sec40'),
('Gradient descent and Ridge', 2, None, '___sec41'),
'and-a-corresponding-example-using-_scikit-learn_'),
('Gradient descent and Ridge',
2,
None,
'gradient-descent-and-ridge'),
('Program example for gradient descent with Ridge Regression',
2,
None,
'___sec42'),
'program-example-for-gradient-descent-with-ridge-regression'),
('Using gradient descent methods, limitations',
2,
None,
'___sec43'),
('Stochastic Gradient Descent', 2, None, '___sec44'),
('Computation of gradients', 2, None, '___sec45'),
('SGD example', 2, None, '___sec46'),
('The gradient step', 2, None, '___sec47'),
('Simple example code', 2, None, '___sec48'),
('When do we stop?', 2, None, '___sec49'),
('Slightly different approach', 2, None, '___sec50'),
('Program for stochastic gradient', 2, None, '___sec51')]}
'using-gradient-descent-methods-limitations'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
('When do we stop?', 2, None, 'when-do-we-stop'),
('Slightly different approach',
2,
None,
'slightly-different-approach'),
('Program for stochastic gradient',
2,
None,
'program-for-stochastic-gradient')]}
end of tocinfo -->
<body>
@@ -175,35 +238,34 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 29, 2020</h4></center> <!-- date -->
<center><h4>Sep 25, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Plan for week 39 </h2>
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<li> Thursday: Repetition of Logistic regression equations and discussion of Gradient methods</li>
<li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
<li> Reading recommendations:</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapter 4</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf" target="_blank">Murphy sections 8.3 and 8.5</a>
See <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank">lecture notes for week 39</a>.
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Thursday September 24 </h2>
<h2 id="thursday-september-30">Thursday September 30 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage" target="_blank">Overview Video, why do we care about gradient methods?</a>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember24.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember24.pdf" target="_blank">link to handwritten notes</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Optimization, the central part of any Machine Learning algortithm </h2>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
<p>
Almost every problem in machine learning and data science starts with
@@ -218,7 +280,7 @@ some approximative/numerical method to compute the minimum.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Revisiting our Logistic Regression case </h2>
<h2 id="revisiting-our-logistic-regression-case">Revisiting our Logistic Regression case </h2>
<p>
In our discussion on Logistic Regression we studied the
@@ -240,7 +302,7 @@ where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">The equations to solve </h2>
<h2 id="the-equations-to-solve">The equations to solve </h2>
<p>
Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \)
@@ -266,7 +328,7 @@ This defines what is called the Hessian matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Solving using Newton-Raphson's method </h2>
<h2 id="solving-using-newton-raphson-s-method">Solving using Newton-Raphson's method </h2>
<p>
If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.
@@ -292,7 +354,7 @@ If we can compute these matrices, in particular the Hessian, the above is often
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Brief reminder on Newton-Raphson's method </h2>
<h2 id="brief-reminder-on-newton-raphson-s-method">Brief reminder on Newton-Raphson's method </h2>
<p>
Let us quickly remind ourselves how we derive the above method.
@@ -309,7 +371,7 @@ normally discourage the use of this method.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">The equations </h2>
<h2 id="the-equations">The equations </h2>
<p>
The Newton-Raphson formula consists geometrically of extending the
@@ -345,7 +407,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Simple geometric interpretation </h2>
<h2 id="simple-geometric-interpretation">Simple geometric interpretation </h2>
<p>
The above is Newton-Raphson's method. It has a simple geometric
@@ -363,7 +425,7 @@ vanishes, then Newton-Raphson may fail totally
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Extending to more than one variable </h2>
<h2 id="extending-to-more-than-one-variable">Extending to more than one variable </h2>
<p>
Newton's method can be generalized to systems of several non-linear equations
@@ -418,7 +480,7 @@ more than two non-linear equations. In our case, the Jacobian matrix is given by
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Steepest descent </h2>
<h2 id="steepest-descent">Steepest descent </h2>
<p>
The basic idea of gradient descent is
@@ -442,7 +504,7 @@ we are always moving towards smaller function values, i.e a minimum.
<p>
<!-- !split -->
<h2 id="___sec11">More on Steepest descent </h2>
<h2 id="more-on-steepest-descent">More on Steepest descent </h2>
<p>
The previous observation is the basis of the method of steepest
@@ -461,7 +523,7 @@ the learning rate within the context of Machine Learning.
<p>
<!-- !split -->
<h2 id="___sec12">The ideal </h2>
<h2 id="the-ideal">The ideal </h2>
<p>
Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global
@@ -487,7 +549,7 @@ Note that the gradient is a function of \( \mathbf{x} =
<p>
<!-- !split -->
<h2 id="___sec13">The sensitiveness of the gradient descent </h2>
<h2 id="the-sensitiveness-of-the-gradient-descent">The sensitiveness of the gradient descent </h2>
<p>
The gradient descent method
@@ -506,7 +568,7 @@ randomness. One such method is that of Stochastic Gradient Descent
<p>
<!-- !split -->
<h2 id="___sec14">Convex functions </h2>
<h2 id="convex-functions">Convex functions </h2>
<p>
Ideally we want our cost/loss function to be convex(concave).
@@ -526,7 +588,7 @@ regular polygons (triangles, rectangles, pentagons, etc...).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Convex function </h2>
<h2 id="convex-function">Convex function </h2>
<p>
<b>Convex function</b>: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.
@@ -534,7 +596,7 @@ regular polygons (triangles, rectangles, pentagons, etc...).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Conditions on convex functions </h2>
<h2 id="conditions-on-convex-functions">Conditions on convex functions </h2>
<p>
In the following we state first and second-order conditions which
@@ -544,7 +606,7 @@ details and proofs we refer to: <a href="http://stanford.edu/boyd/cvxbook/, 2004
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>First order condition.</b>
<b>First order condition</b>
<p>
Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for
all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \)
@@ -559,7 +621,7 @@ note that it is always below the graph.
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Second order condition.</b>
<b>Second order condition</b>
<p>
Assume that \( f \) is twice
differentiable, i.e the Hessian matrix exists at each point in
@@ -576,7 +638,7 @@ This condition is particularly useful since it gives us an procedure for determi
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">More on convex functions </h2>
<h2 id="more-on-convex-functions">More on convex functions </h2>
<p>
The next result is of great importance to us and the reason why we are
@@ -592,7 +654,7 @@ is convex the following result provides invaluable information:
<p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Any minimum is global for convex functions.</b>
<b>Any minimum is global for convex functions</b>
<p>
Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \)
is minimal, where \( f \) is convex and differentiable. Then, any point
@@ -606,7 +668,7 @@ This result means that if we know that the cost/loss function is convex and we a
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Some simple problems </h2>
<h2 id="some-simple-problems">Some simple problems </h2>
<ol>
<li> Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.</li>
@@ -633,15 +695,12 @@ Using the definition of convexity, try to show that a function satisfying the pr
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Friday September 25 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember25.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember25.pdf" target="_blank">link to handwritten notes</a>.
<h2 id="friday-october-1">Friday October 1 </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">Standard steepest descent </h2>
<h2 id="standard-steepest-descent">Standard steepest descent </h2>
<p>
Before we proceed, we would like to discuss the approach called the
@@ -677,7 +736,7 @@ When we have found the exact solution, \( \boldsymbol{r}=0 \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec21">Gradient method </h2>
<h2 id="gradient-method">Gradient method </h2>
<p>
The residual is zero when we reach the minimum of the quadratic equation
@@ -694,7 +753,7 @@ symmetric. This defines also the Hessian and we want it to be positive definit
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">Steepest descent method </h2>
<h2 id="steepest-descent-method">Steepest descent method </h2>
<p>
We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \).
@@ -717,7 +776,7 @@ instead.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Steepest descent method </h2>
<h2 id="steepest-descent-method">Steepest descent method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -747,7 +806,7 @@ and
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Final expressions </h2>
<h2 id="final-expressions">Final expressions </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -790,12 +849,12 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Steepest descent example </h2>
<h2 id="steepest-descent-example">Steepest descent example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy.linalg</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">la</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scipy.optimize</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sopt</span>
@@ -821,7 +880,7 @@ And then as countor plot
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&quot;equal&quot;</span>)
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&quot;equal&quot;</span>)
pt<span style="color: #666666">.</span>contour(xmesh, ymesh, fmesh)
guesses <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">2./5</span>])]
</pre></div>
@@ -830,7 +889,7 @@ Find guesses
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> guesses[<span style="color: #666666">-1</span>]
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>x <span style="color: #666666">=</span> guesses[<span style="color: #666666">-1</span>]
s <span style="color: #666666">=</span> <span style="color: #666666">-</span>df(x)
</pre></div>
<p>
@@ -838,7 +897,7 @@ Run it!
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1d</span>(alpha):
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1d</span>(alpha):
<span style="color: #008000; font-weight: bold">return</span> f(x <span style="color: #666666">+</span> alpha<span style="color: #666666">*</span>s)
alpha_opt <span style="color: #666666">=</span> sopt<span style="color: #666666">.</span>golden(f1d)
@@ -851,7 +910,7 @@ What happened?
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&quot;equal&quot;</span>)
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&quot;equal&quot;</span>)
pt<span style="color: #666666">.</span>contour(xmesh, ymesh, fmesh, <span style="color: #666666">50</span>)
it_array <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array(guesses)
pt<span style="color: #666666">.</span>plot(it_array<span style="color: #666666">.</span>T[<span style="color: #666666">0</span>], it_array<span style="color: #666666">.</span>T[<span style="color: #666666">1</span>], <span style="color: #BA2121">&quot;x-&quot;</span>)
@@ -859,7 +918,7 @@ pt<span style="color: #666666">.</span>plot(it_array<span style="color: #666666"
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -889,7 +948,7 @@ this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -907,7 +966,7 @@ which is zero unless \( i=j \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -934,7 +993,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -966,7 +1025,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Conjugate gradient method and iterations </h2>
<h2 id="conjugate-gradient-method-and-iterations">Conjugate gradient method and iterations </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1002,7 +1061,7 @@ instead.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1032,7 +1091,7 @@ hence the name conjugate gradient method.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec32">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1061,7 +1120,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec33">Conjugate gradient method </h2>
<h2 id="conjugate-gradient-method">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1099,7 +1158,7 @@ $$
<p>
<!-- !split -->
<h2 id="___sec34">Revisiting our first homework </h2>
<h2 id="revisiting-our-first-homework">Revisiting our first homework </h2>
<p>
We will use linear regression as a case study for the gradient descent
@@ -1118,7 +1177,7 @@ We revisit an example similar to what we had in the first homework set. We had a
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(m,<span style="color: #666666">1</span>)
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(m,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(m,<span style="color: #666666">1</span>)
</pre></div>
<p>
@@ -1136,7 +1195,7 @@ $$
<p>
<!-- !split -->
<h2 id="___sec35">Gradient descent example </h2>
<h2 id="gradient-descent-example">Gradient descent example </h2>
<p>
Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
@@ -1161,7 +1220,7 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">The derivative of the cost/loss function </h2>
<h2 id="the-derivative-of-the-cost-loss-function">The derivative of the cost/loss function </h2>
<p>
Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as
@@ -1176,7 +1235,7 @@ where \( X \) is the design matrix defined above.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">The Hessian matrix </h2>
<h2 id="the-hessian-matrix">The Hessian matrix </h2>
The Hessian matrix of \( C(\beta) \) is given by
$$
\boldsymbol{H} \equiv \begin{bmatrix}
@@ -1190,7 +1249,7 @@ This result implies that \( C(\beta) \) is a convex function since the matrix \(
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec38">Simple program </h2>
<h2 id="simple-program">Simple program </h2>
<p>
We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to
@@ -1210,14 +1269,14 @@ And finally we can compare our solution for \( \beta \) with the analytic result
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Gradient Descent Example </h2>
<h2 id="gradient-descent-example">Gradient Descent Example </h2>
<p>
Here our simple example
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
@@ -1266,12 +1325,12 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">And a corresponding example using <b>scikit-learn</b> </h2>
<h2 id="and-a-corresponding-example-using-_scikit-learn_">And a corresponding example using <b>scikit-learn</b> </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
@@ -1291,7 +1350,7 @@ sgdreg<span style="color: #666666">.</span>fit(x,y<span style="color: #666666">.
<p>
<!-- !split -->
<h2 id="___sec41">Gradient descent and Ridge </h2>
<h2 id="gradient-descent-and-ridge">Gradient descent and Ridge </h2>
<p>
We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \beta \),
@@ -1316,11 +1375,11 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec42">Program example for gradient descent with Ridge Regression </h2>
<h2 id="program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> Axes3D
@@ -1367,7 +1426,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec43">Using gradient descent methods, limitations </h2>
<h2 id="using-gradient-descent-methods-limitations">Using gradient descent methods, limitations </h2>
<ul>
<li> <b>Gradient descent (GD) finds local minima of our function</b>. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our cost/loss/risk function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.</li>
@@ -1380,7 +1439,7 @@ plt<span style="color: #666666">.</span>show()
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec44">Stochastic Gradient Descent </h2>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -1398,7 +1457,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec45">Computation of gradients </h2>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>
This in turn means that the gradient can be
@@ -1418,7 +1477,7 @@ minibatches. We denote these minibatches by \( B_k \) where
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec46">SGD example </h2>
<h2 id="sgd-example">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -1442,7 +1501,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec47">The gradient step </h2>
<h2 id="the-gradient-step">The gradient step </h2>
<p>
Thus a gradient descent step now looks like
@@ -1461,12 +1520,12 @@ the number of minibatches, as exemplified in the code below.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">Simple example code </h2>
<h2 id="simple-example-code">Simple example code </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
n <span style="color: #666666">=</span> <span style="color: #666666">100</span> <span style="color: #408080; font-style: italic">#100 datapoints </span>
M <span style="color: #666666">=</span> <span style="color: #666666">5</span> <span style="color: #408080; font-style: italic">#size of each minibatch</span>
@@ -1493,7 +1552,7 @@ all \( n \) datapoints.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">When do we stop? </h2>
<h2 id="when-do-we-stop">When do we stop? </h2>
<p>
A natural question is when do we stop the search for a new minimum?
@@ -1510,7 +1569,7 @@ gave the lowest value.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">Slightly different approach </h2>
<h2 id="slightly-different-approach">Slightly different approach </h2>
<p>
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -1530,7 +1589,7 @@ function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">step_length</span>(t,t0,t1):
<span style="color: #008000; font-weight: bold">return</span> t0<span style="color: #666666">/</span>(t<span style="color: #666666">+</span>t1)
@@ -1558,12 +1617,12 @@ j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Program for stochastic gradient </h2>
<h2 id="program-for-stochastic-gradient">Program for stochastic gradient </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">math</span> <span style="color: #008000; font-weight: bold">import</span> exp, sqrt
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
@@ -1637,7 +1696,7 @@ plt<span style="color: #666666">.</span>show()
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+44 -33
View File
@@ -10,9 +10,9 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 29, 2020**\n",
"Date: **Sep 25, 2021**\n",
"\n",
"Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
@@ -22,14 +22,15 @@
"\n",
"* Friday: Stochastic Gradient descent with examples and automatic differeantion\n",
"\n",
"Reading suggestions for both days: [Aurelien Geron's chapter 4](https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf) and [Murphy sections 8.3 and 8.5](https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf) \n",
"* Reading recommendations:\n",
"\n",
"## Thursday September 24\n",
"See [lecture notes for week 39](https://compphysics.github.io/MachineLearning/doc/web/course.html).\n",
"For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.\n",
"\n",
"## Thursday September 30\n",
"\n",
"[Overview Video, why do we care about gradient methods?](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage)\n",
"\n",
"[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember24.mp4?vrtx=view-as-webpage) and [link to handwritten notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember24.pdf).\n",
"\n",
"\n",
"## Optimization, the central part of any Machine Learning algortithm\n",
"\n",
@@ -552,9 +553,7 @@
"Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).\n",
"\n",
"\n",
"## Friday September 25\n",
"\n",
"[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember25.mp4?vrtx=view-as-webpage) and [link to handwritten notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember25.pdf).\n",
"## Friday October 1\n",
"\n",
"\n",
"## Standard steepest descent\n",
@@ -790,9 +789,10 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -829,9 +829,10 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -849,9 +850,10 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -868,9 +870,10 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -892,9 +895,10 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1270,9 +1274,10 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1439,9 +1444,10 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1502,9 +1508,10 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1588,9 +1595,10 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1768,9 +1776,10 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1832,9 +1841,10 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1873,9 +1883,10 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
doc/src/week39/week39.do.txt
+6 -7
View File
@@ -9,15 +9,16 @@ DATE: today
* Friday: Stochastic Gradient descent with examples and automatic differeantion
Reading suggestions for both days: "Aurelien Geron's chapter 4":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf" and "Murphy sections 8.3 and 8.5":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MachineLearningMurphy.pdf"
* Reading recommendations:
See "lecture notes for week 39":"https://compphysics.github.io/MachineLearning/doc/web/course.html".
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
!split
===== Thursday September 24 =====
===== Thursday September 30 =====
"Overview Video, why do we care about gradient methods?":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage"
"Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember24.mp4?vrtx=view-as-webpage" and "link to handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember24.pdf".
!split
===== Optimization, the central part of any Machine Learning algortithm =====
@@ -391,9 +392,7 @@ Using the definition of convexity, try to show that a function satisfying the pr
!split
===== Friday September 25 =====
"Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember25.mp4?vrtx=view-as-webpage" and "link to handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember25.pdf".
===== Friday October 1 =====
!split