lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week39

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-26 21:56:28 +02:00
parent 7a5f4e8279
commit b393687a01
7 changed files with 63 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/week39-bs.html
+27 -27
View File
@@ -85,7 +85,6 @@ Automatically generated HTML file from DocOnce source
'conditions-on-convex-functions'),
('More on convex functions', 2, None, 'more-on-convex-functions'),
('Some simple problems', 2, None, 'some-simple-problems'),
('Friday October 1', 2, None, 'friday-october-1'),
('Standard steepest descent',
2,
None,
@@ -155,6 +154,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'using-gradient-descent-methods-limitations'),
('Friday October 1', 2, None, 'friday-october-1'),
('Stochastic Gradient Descent',
2,
None,
@@ -228,31 +228,31 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week39-bs017.html#conditions-on-convex-functions" style="font-size: 80%;">Conditions on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs018.html#more-on-convex-functions" style="font-size: 80%;">More on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs019.html#some-simple-problems" style="font-size: 80%;">Some simple problems</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs020.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs021.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs022.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs025.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs026.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs031.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs035.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs038.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#simple-program" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs041.html#and-a-corresponding-example-using-_scikit-learn_" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs020.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs021.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs023.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs023.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs025.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs030.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs038.html#simple-program" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#and-a-corresponding-example-using-_scikit-learn_" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs041.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
@@ -295,7 +295,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 25, 2021</h4></center> <!-- date -->
<center><h4>Sep 26, 2021</h4></center> <!-- date -->
<br>
<p>
doc/pub/week39/html/week39-reveal.html
+8 -8
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 25, 2021</h4></center> <!-- date -->
<center><h4>Sep 26, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -162,8 +162,8 @@ MathJax.Hub.Config({
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<p><li> Thursday: Repetition of Logistic regression equations and discussion of Gradient methods</li>
<p><li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
<p><li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods</li>
<p><li> Friday: Stochastic Gradient descent with examples and automatic differentiation</li>
<p><li> Reading recommendations:</li>
</ul>
<p>
@@ -644,11 +644,6 @@ Using the definition of convexity, try to show that a function satisfying the pr
</section>
<section>
<h2 id="friday-october-1">Friday October 1 </h2>
</section>
<section>
<h2 id="standard-steepest-descent">Standard steepest descent </h2>
@@ -1462,6 +1457,11 @@ plt.show()
</section>
<section>
<h2 id="friday-october-1">Friday October 1 </h2>
</section>
<section>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
doc/pub/week39/html/week39-solarized.html
+9 -9
View File
@@ -105,7 +105,6 @@ div { text-align: justify; text-justify: inter-word; }
'conditions-on-convex-functions'),
('More on convex functions', 2, None, 'more-on-convex-functions'),
('Some simple problems', 2, None, 'some-simple-problems'),
('Friday October 1', 2, None, 'friday-october-1'),
('Standard steepest descent',
2,
None,
@@ -175,6 +174,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'using-gradient-descent-methods-limitations'),
('Friday October 1', 2, None, 'friday-october-1'),
('Stochastic Gradient Descent',
2,
None,
@@ -233,7 +233,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 25, 2021</h4></center> <!-- date -->
<center><h4>Sep 26, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -241,8 +241,8 @@ MathJax.Hub.Config({
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<li> Thursday: Repetition of Logistic regression equations and discussion of Gradient methods</li>
<li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
<li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods</li>
<li> Friday: Stochastic Gradient descent with examples and automatic differentiation</li>
<li> Reading recommendations:</li>
</ul>
@@ -690,11 +690,6 @@ Using the definition of convexity, try to show that a function satisfying the pr
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-october-1">Friday October 1 </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="standard-steepest-descent">Standard steepest descent </h2>
<p>
@@ -1434,6 +1429,11 @@ plt.show()
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-october-1">Friday October 1 </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>
doc/pub/week39/html/week39.html
+9 -9
View File
@@ -110,7 +110,6 @@ div { text-align: justify; text-justify: inter-word; }
'conditions-on-convex-functions'),
('More on convex functions', 2, None, 'more-on-convex-functions'),
('Some simple problems', 2, None, 'some-simple-problems'),
('Friday October 1', 2, None, 'friday-october-1'),
('Standard steepest descent',
2,
None,
@@ -180,6 +179,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'using-gradient-descent-methods-limitations'),
('Friday October 1', 2, None, 'friday-october-1'),
('Stochastic Gradient Descent',
2,
None,
@@ -238,7 +238,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 25, 2021</h4></center> <!-- date -->
<center><h4>Sep 26, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -246,8 +246,8 @@ MathJax.Hub.Config({
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<li> Thursday: Repetition of Logistic regression equations and discussion of Gradient methods</li>
<li> Friday: Stochastic Gradient descent with examples and automatic differeantion</li>
<li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods</li>
<li> Friday: Stochastic Gradient descent with examples and automatic differentiation</li>
<li> Reading recommendations:</li>
</ul>
@@ -695,11 +695,6 @@ Using the definition of convexity, try to show that a function satisfying the pr
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-october-1">Friday October 1 </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="standard-steepest-descent">Standard steepest descent </h2>
<p>
@@ -1439,6 +1434,11 @@ plt<span style="color: #666666">.</span>show()
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="friday-october-1">Friday October 1 </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+6 -4
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 25, 2021**\n",
"Date: **Sep 26, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -18,9 +18,9 @@
"\n",
"## Plan for week 39\n",
"\n",
"* Thursday: Repetition of Logistic regression equations and discussion of Gradient methods\n",
"* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods\n",
"\n",
"* Friday: Stochastic Gradient descent with examples and automatic differeantion\n",
"* Friday: Stochastic Gradient descent with examples and automatic differentiation\n",
"\n",
"* Reading recommendations:\n",
"\n",
@@ -553,7 +553,6 @@
"Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).\n",
"\n",
"\n",
"## Friday October 1\n",
"\n",
"\n",
"## Standard steepest descent\n",
@@ -1665,6 +1664,9 @@
"\n",
"* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.\n",
"\n",
"## Friday October 1\n",
"\n",
"\n",
"## Stochastic Gradient Descent\n",
"\n",
"Stochastic gradient descent (SGD) and variants thereof address some of\n",
doc/src/week39/week39.do.txt
+4 -4
View File
@@ -5,8 +5,8 @@ DATE: today
!split
===== Plan for week 39 =====
* Thursday: Repetition of Logistic regression equations and discussion of Gradient methods
* Friday: Stochastic Gradient descent with examples and automatic differeantion
* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods
* Friday: Stochastic Gradient descent with examples and automatic differentiation
* Reading recommendations:
@@ -391,8 +391,6 @@ o A norm is any function that satisfy the following properties
Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).
!split
===== Friday October 1 =====
!split
@@ -1036,6 +1034,8 @@ plt.show()
* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.
!split
===== Friday October 1 =====
!split