update week 39
This commit is contained in:
Morten Hjorth-Jensen
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "263a45c0",
|
||||
"id": "12bcd5bb",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "155995fb",
|
||||
"id": "cd59f741",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "402606f9",
|
||||
"id": "bdf13ce1",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -36,7 +36,15 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
|
||||
'sections': [('Plan for week 39, September 23-27, 2024',
|
||||
2,
|
||||
None,
|
||||
'plan-for-week-39-september-23-27-2024'),
|
||||
('Lecture Monday September 23',
|
||||
2,
|
||||
None,
|
||||
'lecture-monday-september-23'),
|
||||
('Lab sessions week 39', 2, None, 'lab-sessions-week-39'),
|
||||
('Lecture Monday September 23, Optimization, the central part of '
|
||||
'any Machine Learning algortithm',
|
||||
2,
|
||||
@@ -305,93 +313,95 @@ MathJax.Hub.Config({
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs001.html#plan-for-week-39" style="font-size: 80%;">Plan for week 39</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs002.html#lecture-monday-september-23-optimization-the-central-part-of-any-machine-learning-algortithm" style="font-size: 80%;">Lecture Monday September 23, Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs003.html#revisiting-our-logistic-regression-case" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs004.html#the-equations-to-solve" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs005.html#solving-using-newton-raphson-s-method" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs006.html#brief-reminder-on-newton-raphson-s-method" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs007.html#the-equations" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs008.html#simple-geometric-interpretation" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs009.html#extending-to-more-than-one-variable" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs010.html#steepest-descent" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs011.html#more-on-steepest-descent" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs012.html#the-ideal" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs013.html#the-sensitiveness-of-the-gradient-descent" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs014.html#convex-functions" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs015.html#convex-function" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs016.html#conditions-on-convex-functions" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs017.html#more-on-convex-functions" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs018.html#some-simple-problems" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs019.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs020.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs022.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs022.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs023.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs032.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs032.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs032.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs032.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs029.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs032.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs032.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs032.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs033.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs038.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs035.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs036.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs037.html#simple-program" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs038.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs039.html#and-a-corresponding-example-using-scikit-learn" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs041.html#the-hessian-matrix-for-ridge-regression" style="font-size: 80%;">The Hessian matrix for Ridge Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs042.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs043.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs044.html#improving-gradient-descent-with-momentum" style="font-size: 80%;">Improving gradient descent with momentum</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs082.html#same-code-but-now-with-momentum-gradient-descent" style="font-size: 80%;">Same code but now with momentum gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs046.html#overview-video-on-stochastic-gradient-descent" style="font-size: 80%;">Overview video on Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs047.html#batches-and-mini-batches" style="font-size: 80%;">Batches and mini-batches</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs048.html#stochastic-gradient-descent-sgd" style="font-size: 80%;">Stochastic Gradient Descent (SGD)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs049.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs050.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs051.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs052.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs053.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs054.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs055.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs056.html#time-decay-rate" style="font-size: 80%;">Time decay rate</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs057.html#code-with-a-number-of-minibatches-which-varies" style="font-size: 80%;">Code with a Number of Minibatches which varies</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs058.html#replace-or-not" style="font-size: 80%;">Replace or not</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs059.html#momentum-based-gd" style="font-size: 80%;">Momentum based GD</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs060.html#more-on-momentum-based-approaches" style="font-size: 80%;">More on momentum based approaches</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs061.html#momentum-parameter" style="font-size: 80%;">Momentum parameter</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs062.html#second-moment-of-the-gradient" style="font-size: 80%;">Second moment of the gradient</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs063.html#rms-prop" style="font-size: 80%;">RMS prop</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs064.html#adam-optimizer-https-arxiv-org-abs-1412-6980" style="font-size: 80%;">"ADAM optimizer":"https://arxiv.org/abs/1412.6980"</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs065.html#algorithms-and-codes-for-adagrad-rmsprop-and-adam" style="font-size: 80%;">Algorithms and codes for Adagrad, RMSprop and Adam</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs066.html#practical-tips" style="font-size: 80%;">Practical tips</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs067.html#automatic-differentiation" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs068.html#using-autograd" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs069.html#autograd-with-more-complicated-functions" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs070.html#more-complicated-functions-using-the-elements-of-their-arguments-directly" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs071.html#functions-using-mathematical-functions-from-numpy" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs072.html#more-autograd" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs073.html#and-with-loops" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs074.html#using-recursion" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs075.html#unsupported-functions" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs076.html#the-syntax-a-dot-b-when-finding-the-dot-product" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs077.html#recommended-to-avoid" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs078.html#using-autograd-with-ols" style="font-size: 80%;">Using Autograd with OLS</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs082.html#same-code-but-now-with-momentum-gradient-descent" style="font-size: 80%;">Same code but now with momentum gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs080.html#but-noen-of-these-can-compete-with-newton-s-method" style="font-size: 80%;">But noen of these can compete with Newton's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs081.html#including-stochastic-gradient-descent-with-autograd" style="font-size: 80%;">Including Stochastic Gradient Descent with Autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs082.html#same-code-but-now-with-momentum-gradient-descent" style="font-size: 80%;">Same code but now with momentum gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs083.html#similar-second-order-function-now-problem-but-now-with-adagrad" style="font-size: 80%;">Similar (second order function now) problem but now with AdaGrad</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs084.html#rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent" style="font-size: 80%;">RMSprop for adaptive learning rate with Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs085.html#and-finally-adam-https-arxiv-org-pdf-1412-6980-pdf" style="font-size: 80%;">And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf"</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs086.html#and-logistic-regression" style="font-size: 80%;">And Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs087.html#introducing-jax-https-jax-readthedocs-io-en-latest" style="font-size: 80%;">Introducing "JAX":"https://jax.readthedocs.io/en/latest/"</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs001.html#plan-for-week-39-september-23-27-2024" style="font-size: 80%;">Plan for week 39, September 23-27, 2024</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs002.html#lecture-monday-september-23" style="font-size: 80%;">Lecture Monday September 23</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs003.html#lab-sessions-week-39" style="font-size: 80%;">Lab sessions week 39</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs004.html#lecture-monday-september-23-optimization-the-central-part-of-any-machine-learning-algortithm" style="font-size: 80%;">Lecture Monday September 23, Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs005.html#revisiting-our-logistic-regression-case" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs006.html#the-equations-to-solve" style="font-size: 80%;">The equations to solve</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs007.html#solving-using-newton-raphson-s-method" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs008.html#brief-reminder-on-newton-raphson-s-method" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs009.html#the-equations" style="font-size: 80%;">The equations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs010.html#simple-geometric-interpretation" style="font-size: 80%;">Simple geometric interpretation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs011.html#extending-to-more-than-one-variable" style="font-size: 80%;">Extending to more than one variable</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs012.html#steepest-descent" style="font-size: 80%;">Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs013.html#more-on-steepest-descent" style="font-size: 80%;">More on Steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs014.html#the-ideal" style="font-size: 80%;">The ideal</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs015.html#the-sensitiveness-of-the-gradient-descent" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs016.html#convex-functions" style="font-size: 80%;">Convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs017.html#convex-function" style="font-size: 80%;">Convex function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs018.html#conditions-on-convex-functions" style="font-size: 80%;">Conditions on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs019.html#more-on-convex-functions" style="font-size: 80%;">More on convex functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs020.html#some-simple-problems" style="font-size: 80%;">Some simple problems</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs021.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs022.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs025.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs026.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs031.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs035.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs037.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs038.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs039.html#simple-program" style="font-size: 80%;">Simple program</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs041.html#and-a-corresponding-example-using-scikit-learn" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs042.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs043.html#the-hessian-matrix-for-ridge-regression" style="font-size: 80%;">The Hessian matrix for Ridge Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs044.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs045.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs046.html#improving-gradient-descent-with-momentum" style="font-size: 80%;">Improving gradient descent with momentum</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs084.html#same-code-but-now-with-momentum-gradient-descent" style="font-size: 80%;">Same code but now with momentum gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs048.html#overview-video-on-stochastic-gradient-descent" style="font-size: 80%;">Overview video on Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs049.html#batches-and-mini-batches" style="font-size: 80%;">Batches and mini-batches</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs050.html#stochastic-gradient-descent-sgd" style="font-size: 80%;">Stochastic Gradient Descent (SGD)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs051.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs052.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs053.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs054.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs055.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs056.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs057.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs058.html#time-decay-rate" style="font-size: 80%;">Time decay rate</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs059.html#code-with-a-number-of-minibatches-which-varies" style="font-size: 80%;">Code with a Number of Minibatches which varies</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs060.html#replace-or-not" style="font-size: 80%;">Replace or not</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs061.html#momentum-based-gd" style="font-size: 80%;">Momentum based GD</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs062.html#more-on-momentum-based-approaches" style="font-size: 80%;">More on momentum based approaches</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs063.html#momentum-parameter" style="font-size: 80%;">Momentum parameter</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs064.html#second-moment-of-the-gradient" style="font-size: 80%;">Second moment of the gradient</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs065.html#rms-prop" style="font-size: 80%;">RMS prop</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs066.html#adam-optimizer-https-arxiv-org-abs-1412-6980" style="font-size: 80%;">"ADAM optimizer":"https://arxiv.org/abs/1412.6980"</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs067.html#algorithms-and-codes-for-adagrad-rmsprop-and-adam" style="font-size: 80%;">Algorithms and codes for Adagrad, RMSprop and Adam</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs068.html#practical-tips" style="font-size: 80%;">Practical tips</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs069.html#automatic-differentiation" style="font-size: 80%;">Automatic differentiation</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs070.html#using-autograd" style="font-size: 80%;">Using autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs071.html#autograd-with-more-complicated-functions" style="font-size: 80%;">Autograd with more complicated functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs072.html#more-complicated-functions-using-the-elements-of-their-arguments-directly" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs073.html#functions-using-mathematical-functions-from-numpy" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs074.html#more-autograd" style="font-size: 80%;">More autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs075.html#and-with-loops" style="font-size: 80%;">And with loops</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs076.html#using-recursion" style="font-size: 80%;">Using recursion</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs077.html#unsupported-functions" style="font-size: 80%;">Unsupported functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs078.html#the-syntax-a-dot-b-when-finding-the-dot-product" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs079.html#recommended-to-avoid" style="font-size: 80%;">Recommended to avoid</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs080.html#using-autograd-with-ols" style="font-size: 80%;">Using Autograd with OLS</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs084.html#same-code-but-now-with-momentum-gradient-descent" style="font-size: 80%;">Same code but now with momentum gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs082.html#but-noen-of-these-can-compete-with-newton-s-method" style="font-size: 80%;">But noen of these can compete with Newton's method</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs083.html#including-stochastic-gradient-descent-with-autograd" style="font-size: 80%;">Including Stochastic Gradient Descent with Autograd</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs084.html#same-code-but-now-with-momentum-gradient-descent" style="font-size: 80%;">Same code but now with momentum gradient descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs085.html#similar-second-order-function-now-problem-but-now-with-adagrad" style="font-size: 80%;">Similar (second order function now) problem but now with AdaGrad</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs086.html#rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent" style="font-size: 80%;">RMSprop for adaptive learning rate with Stochastic Gradient Descent</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs087.html#and-finally-adam-https-arxiv-org-pdf-1412-6980-pdf" style="font-size: 80%;">And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf"</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs088.html#and-logistic-regression" style="font-size: 80%;">And Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week39-bs089.html#introducing-jax-https-jax-readthedocs-io-en-latest" style="font-size: 80%;">Introducing "JAX":"https://jax.readthedocs.io/en/latest/"</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -446,7 +456,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._week39-bs008.html">9</a></li>
|
||||
<li><a href="._week39-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._week39-bs087.html">88</a></li>
|
||||
<li><a href="._week39-bs089.html">90</a></li>
|
||||
<li><a href="._week39-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -195,20 +195,22 @@ MathJax.Hub.Config({
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2 id="plan-for-week-39">Plan for week 39 </h2>
|
||||
<h2 id="plan-for-week-39-september-23-27-2024">Plan for week 39, September 23-27, 2024 </h2>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2 id="lecture-monday-september-23">Lecture Monday September 23 </h2>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Material for the lecture on Monday September 23</b>
|
||||
<p>
|
||||
<ul>
|
||||
|
||||
<p><li> Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent</li>
|
||||
<p><li> Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of gradient methods</li>
|
||||
|
||||
<p><li> Stochastic Gradient descent with examples and automatic differentiation</li>
|
||||
|
||||
<p><li> <a href="https://youtu.be/bFRVuIJroHs" target="_blank">Video of lecture</a></li>
|
||||
|
||||
<p><li> Whiteboard notes TBA at <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf</tt></a></li>
|
||||
<p><li> Stochastic Gradient descent with examples and automatic differentiation (theme also for next week).
|
||||
<!-- * <a href="https://youtu.be/bFRVuIJroHs" target="_blank">Video of lecture</a> -->
|
||||
<!-- * Whiteboard notes TBA at <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf</tt></a> --></li>
|
||||
|
||||
<p><li> Readings and Videos:</li>
|
||||
<ul>
|
||||
@@ -217,6 +219,8 @@ MathJax.Hub.Config({
|
||||
|
||||
<p><li> For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.</li>
|
||||
|
||||
<p><li> Raschka et al, pages 53-76 on Logistic regression and pages 37-52 on gradient optimization</li>
|
||||
|
||||
<p><li> <a href="https://www.youtube.com/watch?v=sDv4f4s2SB8" target="_blank">Video on gradient descent</a></li>
|
||||
|
||||
<p><li> <a href="https://www.youtube.com/watch?v=vMh0zPT0tLI" target="_blank">Video on stochastic gradient descent</a></li>
|
||||
@@ -224,7 +228,10 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
</ul>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2 id="lab-sessions-week-39">Lab sessions week 39 </h2>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Material for the active learning sessions on Tuesday and Wednesday</b>
|
||||
|
||||
@@ -63,7 +63,15 @@ div.toc p,a {
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
|
||||
'sections': [('Plan for week 39, September 23-27, 2024',
|
||||
2,
|
||||
None,
|
||||
'plan-for-week-39-september-23-27-2024'),
|
||||
('Lecture Monday September 23',
|
||||
2,
|
||||
None,
|
||||
'lecture-monday-september-23'),
|
||||
('Lab sessions week 39', 2, None, 'lab-sessions-week-39'),
|
||||
('Lecture Monday September 23, Optimization, the central part of '
|
||||
'any Machine Learning algortithm',
|
||||
2,
|
||||
@@ -340,20 +348,24 @@ MathJax.Hub.Config({
|
||||
<br>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="plan-for-week-39">Plan for week 39 </h2>
|
||||
<h2 id="plan-for-week-39-september-23-27-2024">Plan for week 39, September 23-27, 2024 </h2>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="lecture-monday-september-23">Lecture Monday September 23 </h2>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Material for the lecture on Monday September 23</b>
|
||||
<p>
|
||||
<ul>
|
||||
<li> Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent</li>
|
||||
<li> Stochastic Gradient descent with examples and automatic differentiation</li>
|
||||
<li> <a href="https://youtu.be/bFRVuIJroHs" target="_blank">Video of lecture</a></li>
|
||||
<li> Whiteboard notes TBA at <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf</tt></a></li>
|
||||
<li> Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of gradient methods</li>
|
||||
<li> Stochastic Gradient descent with examples and automatic differentiation (theme also for next week).
|
||||
<!-- * <a href="https://youtu.be/bFRVuIJroHs" target="_blank">Video of lecture</a> -->
|
||||
<!-- * Whiteboard notes TBA at <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf</tt></a> --></li>
|
||||
<li> Readings and Videos:</li>
|
||||
<ul>
|
||||
<li> These lecture notes</li>
|
||||
<li> For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.</li>
|
||||
<li> For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.</li>
|
||||
<li> Raschka et al, pages 53-76 on Logistic regression and pages 37-52 on gradient optimization</li>
|
||||
<li> <a href="https://www.youtube.com/watch?v=sDv4f4s2SB8" target="_blank">Video on gradient descent</a></li>
|
||||
<li> <a href="https://www.youtube.com/watch?v=vMh0zPT0tLI" target="_blank">Video on stochastic gradient descent</a></li>
|
||||
</ul>
|
||||
@@ -361,6 +373,9 @@ MathJax.Hub.Config({
|
||||
</div>
|
||||
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="lab-sessions-week-39">Lab sessions week 39 </h2>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Material for the active learning sessions on Tuesday and Wednesday</b>
|
||||
<p>
|
||||
|
||||
@@ -140,7 +140,15 @@ div.toc p,a {
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
|
||||
'sections': [('Plan for week 39, September 23-27, 2024',
|
||||
2,
|
||||
None,
|
||||
'plan-for-week-39-september-23-27-2024'),
|
||||
('Lecture Monday September 23',
|
||||
2,
|
||||
None,
|
||||
'lecture-monday-september-23'),
|
||||
('Lab sessions week 39', 2, None, 'lab-sessions-week-39'),
|
||||
('Lecture Monday September 23, Optimization, the central part of '
|
||||
'any Machine Learning algortithm',
|
||||
2,
|
||||
@@ -417,20 +425,24 @@ MathJax.Hub.Config({
|
||||
<br>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="plan-for-week-39">Plan for week 39 </h2>
|
||||
<h2 id="plan-for-week-39-september-23-27-2024">Plan for week 39, September 23-27, 2024 </h2>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="lecture-monday-september-23">Lecture Monday September 23 </h2>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Material for the lecture on Monday September 23</b>
|
||||
<p>
|
||||
<ul>
|
||||
<li> Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent</li>
|
||||
<li> Stochastic Gradient descent with examples and automatic differentiation</li>
|
||||
<li> <a href="https://youtu.be/bFRVuIJroHs" target="_blank">Video of lecture</a></li>
|
||||
<li> Whiteboard notes TBA at <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf</tt></a></li>
|
||||
<li> Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of gradient methods</li>
|
||||
<li> Stochastic Gradient descent with examples and automatic differentiation (theme also for next week).
|
||||
<!-- * <a href="https://youtu.be/bFRVuIJroHs" target="_blank">Video of lecture</a> -->
|
||||
<!-- * Whiteboard notes TBA at <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf</tt></a> --></li>
|
||||
<li> Readings and Videos:</li>
|
||||
<ul>
|
||||
<li> These lecture notes</li>
|
||||
<li> For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.</li>
|
||||
<li> For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.</li>
|
||||
<li> Raschka et al, pages 53-76 on Logistic regression and pages 37-52 on gradient optimization</li>
|
||||
<li> <a href="https://www.youtube.com/watch?v=sDv4f4s2SB8" target="_blank">Video on gradient descent</a></li>
|
||||
<li> <a href="https://www.youtube.com/watch?v=vMh0zPT0tLI" target="_blank">Video on stochastic gradient descent</a></li>
|
||||
</ul>
|
||||
@@ -438,6 +450,9 @@ MathJax.Hub.Config({
|
||||
</div>
|
||||
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="lab-sessions-week-39">Lab sessions week 39 </h2>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Material for the active learning sessions on Tuesday and Wednesday</b>
|
||||
<p>
|
||||
|
||||
Binary file not shown.
+312
-292
File diff suppressed because it is too large
Load Diff
@@ -3,23 +3,28 @@ AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of
|
||||
DATE: Week 39
|
||||
|
||||
!split
|
||||
===== Plan for week 39 =====
|
||||
===== Plan for week 39, September 23-27, 2024 =====
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== Lecture Monday September 23 =====
|
||||
|
||||
!bblock Material for the lecture on Monday September 23
|
||||
* Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent
|
||||
* Stochastic Gradient descent with examples and automatic differentiation
|
||||
* "Video of lecture":"https://youtu.be/bFRVuIJroHs"
|
||||
* Whiteboard notes TBA at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf"
|
||||
* Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of gradient methods
|
||||
* Stochastic Gradient descent with examples and automatic differentiation (theme also for next week).
|
||||
# * "Video of lecture":"https://youtu.be/bFRVuIJroHs"
|
||||
# * Whiteboard notes TBA at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf"
|
||||
* Readings and Videos:
|
||||
* These lecture notes
|
||||
* For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.
|
||||
* For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and sections 8.3-8.6. We will come back to the latter chapter in our discussion of Neural networks as well.
|
||||
* Raschka et al, pages 53-76 on Logistic regression and pages 37-52 on gradient optimization
|
||||
* "Video on gradient descent":"https://www.youtube.com/watch?v=sDv4f4s2SB8"
|
||||
* "Video on stochastic gradient descent":"https://www.youtube.com/watch?v=vMh0zPT0tLI"
|
||||
!eblock
|
||||
|
||||
!split
|
||||
===== Lab sessions week 39 =====
|
||||
|
||||
!bblock Material for the active learning sessions on Tuesday and Wednesday
|
||||
* Discussions on how to structure your report for the first project
|
||||
|
||||
Reference in New Issue
Block a user