lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

typos and cleaning up week39

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-29 11:13:39 +02:00
parent 2d036c7351
commit 57c1fcee57
7 changed files with 221 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/week39-bs.html
+51 -49
View File
@@ -59,6 +59,7 @@ Automatically generated HTML file from DocOnce source
None,
'revisiting-our-logistic-regression-case'),
('The equations to solve', 2, None, 'the-equations-to-solve'),
('To be added', 2, None, 'to-be-added'),
("Solving using Newton-Raphson's method",
2,
None,
@@ -223,53 +224,54 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week39-bs006.html#optimization-the-central-part-of-any-machine-learning-algortithm" style="font-size: 80%;">Optimization, the central part of any Machine Learning algortithm</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs007.html#revisiting-our-logistic-regression-case" style="font-size: 80%;">Revisiting our Logistic Regression case</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs008.html#the-equations-to-solve" style="font-size: 80%;">The equations to solve</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs009.html#solving-using-newton-raphson-s-method" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs010.html#brief-reminder-on-newton-raphson-s-method" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs011.html#the-equations" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs012.html#simple-geometric-interpretation" style="font-size: 80%;">Simple geometric interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs013.html#extending-to-more-than-one-variable" style="font-size: 80%;">Extending to more than one variable</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs014.html#steepest-descent" style="font-size: 80%;">Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs015.html#more-on-steepest-descent" style="font-size: 80%;">More on Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs016.html#the-ideal" style="font-size: 80%;">The ideal</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs017.html#the-sensitiveness-of-the-gradient-descent" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs018.html#convex-functions" style="font-size: 80%;">Convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs019.html#convex-function" style="font-size: 80%;">Convex function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs020.html#conditions-on-convex-functions" style="font-size: 80%;">Conditions on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs021.html#more-on-convex-functions" style="font-size: 80%;">More on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs022.html#some-simple-problems" style="font-size: 80%;">Some simple problems</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs023.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs026.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs026.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs027.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs028.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs033.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs036.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs039.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs041.html#simple-program" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#and-a-corresponding-example-using-_scikit-learn_" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs048.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs049.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs052.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs053.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs054.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs055.html#program-for-stochastic-gradient" style="font-size: 80%;">Program for stochastic gradient</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs009.html#to-be-added" style="font-size: 80%;">To be added</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs010.html#solving-using-newton-raphson-s-method" style="font-size: 80%;">Solving using Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs011.html#brief-reminder-on-newton-raphson-s-method" style="font-size: 80%;">Brief reminder on Newton-Raphson's method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs012.html#the-equations" style="font-size: 80%;">The equations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs013.html#simple-geometric-interpretation" style="font-size: 80%;">Simple geometric interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs014.html#extending-to-more-than-one-variable" style="font-size: 80%;">Extending to more than one variable</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs015.html#steepest-descent" style="font-size: 80%;">Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs016.html#more-on-steepest-descent" style="font-size: 80%;">More on Steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs017.html#the-ideal" style="font-size: 80%;">The ideal</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs018.html#the-sensitiveness-of-the-gradient-descent" style="font-size: 80%;">The sensitiveness of the gradient descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs019.html#convex-functions" style="font-size: 80%;">Convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs020.html#convex-function" style="font-size: 80%;">Convex function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs021.html#conditions-on-convex-functions" style="font-size: 80%;">Conditions on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs022.html#more-on-convex-functions" style="font-size: 80%;">More on convex functions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs023.html#some-simple-problems" style="font-size: 80%;">Some simple problems</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs024.html#standard-steepest-descent" style="font-size: 80%;">Standard steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs025.html#gradient-method" style="font-size: 80%;">Gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs027.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs027.html#steepest-descent-method" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs028.html#final-expressions" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs029.html#steepest-descent-example" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs034.html#conjugate-gradient-method-and-iterations" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs037.html#conjugate-gradient-method" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs038.html#revisiting-our-first-homework" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#gradient-descent-example" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs040.html#the-derivative-of-the-cost-loss-function" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs041.html#the-hessian-matrix" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs042.html#simple-program" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#gradient-descent-example" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#and-a-corresponding-example-using-_scikit-learn_" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#gradient-descent-and-ridge" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs048.html#friday-october-1" style="font-size: 80%;">Friday October 1</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs049.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs052.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs053.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs054.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs055.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs056.html#program-for-stochastic-gradient" style="font-size: 80%;">Program for stochastic gradient</a></li>
</ul>
</li>
@@ -304,7 +306,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 28, 2021</h4></center> <!-- date -->
<center><h4>Sep 29, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -328,7 +330,7 @@ MathJax.Hub.Config({
<li><a href="._week39-bs008.html">9</a></li>
<li><a href="._week39-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week39-bs055.html">56</a></li>
<li><a href="._week39-bs056.html">57</a></li>
<li><a href="._week39-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week39/html/week39-reveal.html
+21 -4
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 28, 2021</h4></center> <!-- date -->
<center><h4>Sep 29, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -241,7 +241,7 @@ plt.legend()
plt.show()
</pre></div>
<p>
Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function <b>RidgeCV</b> instead of just calling the <b>Ridge</b> function. For <b>RidgeCV</b> we need to passe the array of \( \lambda \) values.
Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function <b>RidgeCV</b> instead of just calling the <b>Ridge</b> function. For <b>RidgeCV</b> we need to pass the array of \( \lambda \) values.
By inspecting the figure we can in turn determine which is the optimal regularization parameter.
This becomes however less functional in the long run.
</section>
@@ -302,7 +302,10 @@ ypredictRidge = gridsearch.predict(X_test)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;R2 score: {</span>R2(y_test,ypredictRidge)<span style="color: #CD5555">}&quot;</span>)
</pre></div>
<p>
By default the grid search function includes cross validation with five folds. The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" target="_blank">Scikit-Learn documentation</a> contains more information on how to set the different parameters.
By default the grid search function includes cross validation with
five folds. The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" target="_blank">Scikit-Learn
documentation</a>
contains more information on how to set the different parameters.
<p>
If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit.
@@ -317,7 +320,10 @@ An alternative to the above manual grid set up, is to use a random
search where the parameters are tuned from a random distribution
(uniform below) for a fixed number of iterations. A model is
constructed and evaluated for each combination of chosen parameters.
We repeat the previous example but now with a random search.
We repeat the previous example but now with a random search. Note
that values of \( \lambda \) are now limited to be within \( x\in
[0,1] \). This domain may not be the most relevant one for the specific
case under study.
<p>
@@ -438,6 +444,17 @@ This defines what is called the Hessian matrix.
</section>
<section>
<h2 id="to-be-added">To be added </h2>
<p>
We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix.
<p>
Make link with linear regression and the Hessian matrix from linear regression.
</section>
<section>
<h2 id="solving-using-newton-raphson-s-method">Solving using Newton-Raphson's method </h2>
doc/pub/week39/html/week39-solarized.html
+22 -4
View File
@@ -79,6 +79,7 @@ div { text-align: justify; text-justify: inter-word; }
None,
'revisiting-our-logistic-regression-case'),
('The equations to solve', 2, None, 'the-equations-to-solve'),
('To be added', 2, None, 'to-be-added'),
("Solving using Newton-Raphson's method",
2,
None,
@@ -239,7 +240,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 28, 2021</h4></center> <!-- date -->
<center><h4>Sep 29, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -325,7 +326,7 @@ plt.legend()
plt.show()
</pre></div>
<p>
Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function <b>RidgeCV</b> instead of just calling the <b>Ridge</b> function. For <b>RidgeCV</b> we need to passe the array of \( \lambda \) values.
Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function <b>RidgeCV</b> instead of just calling the <b>Ridge</b> function. For <b>RidgeCV</b> we need to pass the array of \( \lambda \) values.
By inspecting the figure we can in turn determine which is the optimal regularization parameter.
This becomes however less functional in the long run.
@@ -386,7 +387,10 @@ ypredictRidge = gridsearch.predict(X_test)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;R2 score: {</span>R2(y_test,ypredictRidge)<span style="color: #CD5555">}&quot;</span>)
</pre></div>
<p>
By default the grid search function includes cross validation with five folds. The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" target="_blank">Scikit-Learn documentation</a> contains more information on how to set the different parameters.
By default the grid search function includes cross validation with
five folds. The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" target="_blank">Scikit-Learn
documentation</a>
contains more information on how to set the different parameters.
<p>
If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit.
@@ -401,7 +405,10 @@ An alternative to the above manual grid set up, is to use a random
search where the parameters are tuned from a random distribution
(uniform below) for a fixed number of iterations. A model is
constructed and evaluated for each combination of chosen parameters.
We repeat the previous example but now with a random search.
We repeat the previous example but now with a random search. Note
that values of \( \lambda \) are now limited to be within \( x\in
[0,1] \). This domain may not be the most relevant one for the specific
case under study.
<p>
@@ -516,6 +523,17 @@ This defines what is called the Hessian matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="to-be-added">To be added </h2>
<p>
We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix.
<p>
Make link with linear regression and the Hessian matrix from linear regression.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="solving-using-newton-raphson-s-method">Solving using Newton-Raphson's method </h2>
<p>
doc/pub/week39/html/week39.html
+22 -4
View File
@@ -84,6 +84,7 @@ div { text-align: justify; text-justify: inter-word; }
None,
'revisiting-our-logistic-regression-case'),
('The equations to solve', 2, None, 'the-equations-to-solve'),
('To be added', 2, None, 'to-be-added'),
("Solving using Newton-Raphson's method",
2,
None,
@@ -244,7 +245,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 28, 2021</h4></center> <!-- date -->
<center><h4>Sep 29, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -330,7 +331,7 @@ plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function <b>RidgeCV</b> instead of just calling the <b>Ridge</b> function. For <b>RidgeCV</b> we need to passe the array of \( \lambda \) values.
Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function <b>RidgeCV</b> instead of just calling the <b>Ridge</b> function. For <b>RidgeCV</b> we need to pass the array of \( \lambda \) values.
By inspecting the figure we can in turn determine which is the optimal regularization parameter.
This becomes however less functional in the long run.
@@ -391,7 +392,10 @@ ypredictRidge <span style="color: #666666">=</span> gridsearch<span style="color
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;R2 score: </span><span style="color: #BB6688; font-weight: bold">{</span>R2(y_test,ypredictRidge)<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&quot;</span>)
</pre></div>
<p>
By default the grid search function includes cross validation with five folds. The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" target="_blank">Scikit-Learn documentation</a> contains more information on how to set the different parameters.
By default the grid search function includes cross validation with
five folds. The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" target="_blank">Scikit-Learn
documentation</a>
contains more information on how to set the different parameters.
<p>
If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit.
@@ -406,7 +410,10 @@ An alternative to the above manual grid set up, is to use a random
search where the parameters are tuned from a random distribution
(uniform below) for a fixed number of iterations. A model is
constructed and evaluated for each combination of chosen parameters.
We repeat the previous example but now with a random search.
We repeat the previous example but now with a random search. Note
that values of \( \lambda \) are now limited to be within \( x\in
[0,1] \). This domain may not be the most relevant one for the specific
case under study.
<p>
@@ -521,6 +528,17 @@ This defines what is called the Hessian matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="to-be-added">To be added </h2>
<p>
We will add here an example which computes the likelihood \( p_i \), sets up the gradient and the Hessian matrix.
<p>
Make link with linear regression and the Hessian matrix from linear regression.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="solving-using-newton-raphson-s-method">Solving using Newton-Raphson's method </h2>
<p>
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+86 -83
View File
File diff suppressed because one or more lines are too long
doc/src/week39/week39.do.txt
+19 -4
View File
@@ -78,7 +78,7 @@ plt.show()
!ec
Here we have performed a rather data greedy calculation as function of the regularization parameter $\lambda$. There is no resampling here. The latter can easily be added by employing the function _RidgeCV_ instead of just calling the _Ridge_ function. For _RidgeCV_ we need to passe the array of $\lambda$ values.
Here we have performed a rather data greedy calculation as function of the regularization parameter $\lambda$. There is no resampling here. The latter can easily be added by employing the function _RidgeCV_ instead of just calling the _Ridge_ function. For _RidgeCV_ we need to pass the array of $\lambda$ values.
By inspecting the figure we can in turn determine which is the optimal regularization parameter.
This becomes however less functional in the long run.
@@ -137,7 +137,10 @@ print(f"R2 score: {R2(y_test,ypredictRidge)}")
!ec
By default the grid search function includes cross validation with five folds. The "Scikit-Learn documentation":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" contains more information on how to set the different parameters.
By default the grid search function includes cross validation with
five folds. The "Scikit-Learn
documentation":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV"
contains more information on how to set the different parameters.
If we take out the random noise, running the above codes results in $\lambda=0$ yielding the best fit.
@@ -149,8 +152,10 @@ An alternative to the above manual grid set up, is to use a random
search where the parameters are tuned from a random distribution
(uniform below) for a fixed number of iterations. A model is
constructed and evaluated for each combination of chosen parameters.
We repeat the previous example but now with a random search.
We repeat the previous example but now with a random search. Note
that values of $\lambda$ are now limited to be within $x\in
[0,1]$. This domain may not be the most relevant one for the specific
case under study.
!bc pycod
@@ -258,6 +263,16 @@ $p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compa
!et
This defines what is called the Hessian matrix.
!split
===== To be added =====
We will add here an example which computes the likelihood $p_i$, sets up the gradient and the Hessian matrix.
Make link with linear regression and the Hessian matrix from linear regression.
!split
===== Solving using Newton-Raphson's method =====