lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added code on grid search

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-27 17:32:18 +02:00
parent b393687a01
commit 1b74711189
7 changed files with 150 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/week39-bs.html
+57 -52
View File
@@ -43,6 +43,10 @@ Automatically generated HTML file from DocOnce source
{'highest level': 2,
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
('Thursday September 30', 2, None, 'thursday-september-30'),
('Searching for Optimal Regularization Parameters $\\lambda$',
2,
None,
'searching-for-optimal-regularization-parameters-lambda'),
('Optimization, the central part of any Machine Learning '
'algortithm',
2,
@@ -211,56 +215,57 @@ MathJax.Hub.Config({
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week39-bs001.html#plan-for-week-39" style="font-size: 80%;">Plan for week 39</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs002.html#thursday-september-30" style="font-size: 80%;">Thursday September 30</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs003.html#optimization-the-central-part-of-any-machine-learning-algortithm" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs004.html#revisiting-our-logistic-regression-case" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs005.html#the-equations-to-solve" style="font-size: 80%;">The equations to solve</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs006.html#solving-using-newton-raphson-s-method" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs007.html#brief-reminder-on-newton-raphson-s-method" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs008.html#the-equations" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs009.html#simple-geometric-interpretation" style="font-size: 80%;">Simple geometric interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs010.html#extending-to-more-than-one-variable" style="font-size: 80%;">Extending to more than one variable</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs011.html#steepest-descent" style="font-size: 80%;">Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs012.html#more-on-steepest-descent" style="font-size: 80%;">More on Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs013.html#the-ideal" style="font-size: 80%;">The ideal</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs014.html#the-sensitiveness-of-the-gradient-descent" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs015.html#convex-functions" style="font-size: 80%;">Convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs016.html#convex-function" style="font-size: 80%;">Convex function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs017.html#conditions-on-convex-functions" style="font-size: 80%;">Conditions on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs018.html#more-on-convex-functions" style="font-size: 80%;">More on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs019.html#some-simple-problems" style="font-size: 80%;">Some simple problems</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs020.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs021.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs023.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs023.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs025.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs030.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs038.html#simple-program" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#and-a-corresponding-example-using-_scikit-learn_" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs041.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs048.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs049.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs052.html#program-for-stochastic-gradient" style="font-size: 80%;">Program for stochastic gradient</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs003.html#searching-for-optimal-regularization-parameters-lambda" style="font-size: 80%;">Searching for Optimal Regularization Parameters \( \lambda \)</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs004.html#optimization-the-central-part-of-any-machine-learning-algortithm" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs005.html#revisiting-our-logistic-regression-case" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs006.html#the-equations-to-solve" style="font-size: 80%;">The equations to solve</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs007.html#solving-using-newton-raphson-s-method" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs008.html#brief-reminder-on-newton-raphson-s-method" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs009.html#the-equations" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs010.html#simple-geometric-interpretation" style="font-size: 80%;">Simple geometric interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs011.html#extending-to-more-than-one-variable" style="font-size: 80%;">Extending to more than one variable</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs012.html#steepest-descent" style="font-size: 80%;">Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs013.html#more-on-steepest-descent" style="font-size: 80%;">More on Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs014.html#the-ideal" style="font-size: 80%;">The ideal</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs015.html#the-sensitiveness-of-the-gradient-descent" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs016.html#convex-functions" style="font-size: 80%;">Convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs017.html#convex-function" style="font-size: 80%;">Convex function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs018.html#conditions-on-convex-functions" style="font-size: 80%;">Conditions on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs019.html#more-on-convex-functions" style="font-size: 80%;">More on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs020.html#some-simple-problems" style="font-size: 80%;">Some simple problems</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs021.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs022.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs025.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs026.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs031.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs035.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs038.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#simple-program" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs041.html#and-a-corresponding-example-using-_scikit-learn_" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs048.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs049.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs052.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs053.html#program-for-stochastic-gradient" style="font-size: 80%;">Program for stochastic gradient</a></li>
</ul>
</li>
@@ -295,7 +300,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 26, 2021</h4></center> <!-- date -->
<center><h4>Sep 27, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -319,7 +324,7 @@ MathJax.Hub.Config({
<li><a href="._week39-bs008.html">9</a></li>
<li><a href="._week39-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week39-bs052.html">53</a></li>
<li><a href="._week39-bs053.html">54</a></li>
<li><a href="._week39-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week39/html/week39-reveal.html
+17 -2
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 26, 2021</h4></center> <!-- date -->
<center><h4>Sep 27, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -162,7 +162,7 @@ MathJax.Hub.Config({
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<p><li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods</li>
<p><li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Discussion of project 1</li>
<p><li> Friday: Stochastic Gradient descent with examples and automatic differentiation</li>
<p><li> Reading recommendations:</li>
</ul>
@@ -181,6 +181,21 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5
</section>
<section>
<h2 id="searching-for-optimal-regularization-parameters-lambda">Searching for Optimal Regularization Parameters \( \lambda \) </h2>
<p>
In project 1 when using Ridge and Lasso regression, we end up
searching for the optimal parameter \( \lambda \) which minimizes our
selected scores (MSE or \( R2 \) values for example). The brute force
approach, as discussed in the code here for Ridge regression consists
in evaluating the MSE as function of different \( \lambda \) values.
<p>
An alternative is to use the so-called grid search functionality included with the library <b>Scikit-Learn</b>, as demonstrated for the same example here.
</section>
<section>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
doc/pub/week39/html/week39-solarized.html
+21 -2
View File
@@ -63,6 +63,10 @@ div { text-align: justify; text-justify: inter-word; }
{'highest level': 2,
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
('Thursday September 30', 2, None, 'thursday-september-30'),
('Searching for Optimal Regularization Parameters $\\lambda$',
2,
None,
'searching-for-optimal-regularization-parameters-lambda'),
('Optimization, the central part of any Machine Learning '
'algortithm',
2,
@@ -233,7 +237,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 26, 2021</h4></center> <!-- date -->
<center><h4>Sep 27, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -241,7 +245,7 @@ MathJax.Hub.Config({
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods</li>
<li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Discussion of project 1</li>
<li> Friday: Stochastic Gradient descent with examples and automatic differentiation</li>
<li> Reading recommendations:</li>
</ul>
@@ -260,6 +264,21 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="searching-for-optimal-regularization-parameters-lambda">Searching for Optimal Regularization Parameters \( \lambda \) </h2>
<p>
In project 1 when using Ridge and Lasso regression, we end up
searching for the optimal parameter \( \lambda \) which minimizes our
selected scores (MSE or \( R2 \) values for example). The brute force
approach, as discussed in the code here for Ridge regression consists
in evaluating the MSE as function of different \( \lambda \) values.
<p>
An alternative is to use the so-called grid search functionality included with the library <b>Scikit-Learn</b>, as demonstrated for the same example here.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
<p>
doc/pub/week39/html/week39.html
+21 -2
View File
@@ -68,6 +68,10 @@ div { text-align: justify; text-justify: inter-word; }
{'highest level': 2,
'sections': [('Plan for week 39', 2, None, 'plan-for-week-39'),
('Thursday September 30', 2, None, 'thursday-september-30'),
('Searching for Optimal Regularization Parameters $\\lambda$',
2,
None,
'searching-for-optimal-regularization-parameters-lambda'),
('Optimization, the central part of any Machine Learning '
'algortithm',
2,
@@ -238,7 +242,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 26, 2021</h4></center> <!-- date -->
<center><h4>Sep 27, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -246,7 +250,7 @@ MathJax.Hub.Config({
<h2 id="plan-for-week-39">Plan for week 39 </h2>
<ul>
<li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods</li>
<li> Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Discussion of project 1</li>
<li> Friday: Stochastic Gradient descent with examples and automatic differentiation</li>
<li> Reading recommendations:</li>
</ul>
@@ -265,6 +269,21 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="searching-for-optimal-regularization-parameters-lambda">Searching for Optimal Regularization Parameters \( \lambda \) </h2>
<p>
In project 1 when using Ridge and Lasso regression, we end up
searching for the optimal parameter \( \lambda \) which minimizes our
selected scores (MSE or \( R2 \) values for example). The brute force
approach, as discussed in the code here for Ridge regression consists
in evaluating the MSE as function of different \( \lambda \) values.
<p>
An alternative is to use the so-called grid search functionality included with the library <b>Scikit-Learn</b>, as demonstrated for the same example here.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
<p>
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+17 -2
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 26, 2021**\n",
"Date: **Sep 27, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -18,7 +18,7 @@
"\n",
"## Plan for week 39\n",
"\n",
"* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods\n",
"* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Discussion of project 1\n",
"\n",
"* Friday: Stochastic Gradient descent with examples and automatic differentiation\n",
"\n",
@@ -32,6 +32,21 @@
"[Overview Video, why do we care about gradient methods?](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage)\n",
"\n",
"\n",
"## Searching for Optimal Regularization Parameters $\\lambda$\n",
"\n",
"In project 1 when using Ridge and Lasso regression, we end up\n",
"searching for the optimal parameter $\\lambda$ which minimizes our\n",
"selected scores (MSE or $R2$ values for example). The brute force\n",
"approach, as discussed in the code here for Ridge regression consists\n",
"in evaluating the MSE as function of different $\\lambda$ values.\n",
"\n",
"An alternative is to use the so-called grid search functionality included with the library **Scikit-Learn**, as demonstrated for the same example here.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Optimization, the central part of any Machine Learning algortithm\n",
"\n",
"Almost every problem in machine learning and data science starts with\n",
doc/src/week39/week39.do.txt
+17 -1
View File
@@ -5,7 +5,7 @@ DATE: today
!split
===== Plan for week 39 =====
* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods
* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Discussion of project 1
* Friday: Stochastic Gradient descent with examples and automatic differentiation
@@ -20,6 +20,22 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5
"Overview Video, why do we care about gradient methods?":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage"
!split
===== Searching for Optimal Regularization Parameters $\lambda$ =====
In project 1 when using Ridge and Lasso regression, we end up
searching for the optimal parameter $\lambda$ which minimizes our
selected scores (MSE or $R2$ values for example). The brute force
approach, as discussed in the code here for Ridge regression consists
in evaluating the MSE as function of different $\lambda$ values.
An alternative is to use the so-called grid search functionality included with the library _Scikit-Learn_, as demonstrated for the same example here.
!split
===== Optimization, the central part of any Machine Learning algortithm =====