lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update project 2

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-10-09 06:03:15 +02:00
parent 9923bb2339
commit 3c553e703e
9 changed files with 69 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Projects/2023/Project2/html/._Project2-bs000.html
+8 -7
View File
@@ -145,7 +145,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Oct 8, 2023</h4>
<h4>Oct 9, 2023</h4>
</center> <!-- date -->
<br>
@@ -205,16 +205,18 @@ function.
<p>You should include in your analysis of the GD and SGD codes the following elements</p>
<ol>
<li> A plain gradient descent with a fixed learning rate (you will need to tune it).</li>
<li> Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).</li>
<li> Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)</li>
<li> A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.</li>
<li> Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.</li>
<li> Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.</li>
<li> Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.</li>
<li> Add RMSprop and Adam to your library of methods for tuning the learning rate.</li>
</ol>
<p>The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" target="_self">week 39 contain more
<p>The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" target="_self">weeks 39 and 40contain more
details</a> and code examples. Feel free to use these examples.
</p>
<ol>
<li> Replace thereafter your analytical gradient with either <b>Autograd</b> or <b>JAX</b></li>
</ol>
<p>In summary, you should
perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
@@ -340,7 +342,6 @@ your final project 3, but now applied to other data sets.
<ol>
<li> The text of Michael Nielsen is highly recommended, see <a href="http://neuralnetworksanddeeplearning.com/" target="_self">Nielsen's book</a>. It is an excellent read.</li>
<li> The textbook of <a href="https://www.springer.com/gp/book/9780387848570" target="_self">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>, chapters 3 and 7 are the most relevant ones for the analysis here.</li>
<li> <a href="https://arxiv.org/abs/1803.08823" target="_self">Mehta et al, arXiv 1803.08823</a>, <em>A high-bias, low-variance introduction to Machine Learning for physicists</em>, ArXiv:1803.08823.</li>
</ol>
<li> <a href="https://www.deeplearningbook.org/" target="_self">Goodfellow, Bengio and Courville</a>, <em>Deep Learning</em>.
doc/Projects/2023/Project2/html/Project2-bs.html
+8 -7
View File
@@ -145,7 +145,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Oct 8, 2023</h4>
<h4>Oct 9, 2023</h4>
</center> <!-- date -->
<br>
@@ -205,16 +205,18 @@ function.
<p>You should include in your analysis of the GD and SGD codes the following elements</p>
<ol>
<li> A plain gradient descent with a fixed learning rate (you will need to tune it).</li>
<li> Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).</li>
<li> Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)</li>
<li> A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.</li>
<li> Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.</li>
<li> Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.</li>
<li> Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.</li>
<li> Add RMSprop and Adam to your library of methods for tuning the learning rate.</li>
</ol>
<p>The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" target="_self">week 39 contain more
<p>The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" target="_self">weeks 39 and 40contain more
details</a> and code examples. Feel free to use these examples.
</p>
<ol>
<li> Replace thereafter your analytical gradient with either <b>Autograd</b> or <b>JAX</b></li>
</ol>
<p>In summary, you should
perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
@@ -340,7 +342,6 @@ your final project 3, but now applied to other data sets.
<ol>
<li> The text of Michael Nielsen is highly recommended, see <a href="http://neuralnetworksanddeeplearning.com/" target="_self">Nielsen's book</a>. It is an excellent read.</li>
<li> The textbook of <a href="https://www.springer.com/gp/book/9780387848570" target="_self">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>, chapters 3 and 7 are the most relevant ones for the analysis here.</li>
<li> <a href="https://arxiv.org/abs/1803.08823" target="_self">Mehta et al, arXiv 1803.08823</a>, <em>A high-bias, low-variance introduction to Machine Learning for physicists</em>, ArXiv:1803.08823.</li>
</ol>
<li> <a href="https://www.deeplearningbook.org/" target="_self">Goodfellow, Bengio and Courville</a>, <em>Deep Learning</em>.
doc/Projects/2023/Project2/html/Project2.html
+8 -7
View File
@@ -187,7 +187,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Oct 8, 2023</h4>
<h4>Oct 9, 2023</h4>
</center> <!-- date -->
<br>
<h2 id="classification-and-regression-from-linear-and-logistic-regression-to-neural-networks">Classification and Regression, from linear and logistic regression to neural networks </h2>
@@ -244,16 +244,18 @@ function.
<p>You should include in your analysis of the GD and SGD codes the following elements</p>
<ol>
<li> A plain gradient descent with a fixed learning rate (you will need to tune it).</li>
<li> Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).</li>
<li> Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)</li>
<li> A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.</li>
<li> Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.</li>
<li> Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.</li>
<li> Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.</li>
<li> Add RMSprop and Adam to your library of methods for tuning the learning rate.</li>
</ol>
<p>The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" target="_blank">week 39 contain more
<p>The lecture notes from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" target="_blank">weeks 39 and 40contain more
details</a> and code examples. Feel free to use these examples.
</p>
<ol>
<li> Replace thereafter your analytical gradient with either <b>Autograd</b> or <b>JAX</b></li>
</ol>
<p>In summary, you should
perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
@@ -379,7 +381,6 @@ your final project 3, but now applied to other data sets.
<ol>
<li> The text of Michael Nielsen is highly recommended, see <a href="http://neuralnetworksanddeeplearning.com/" target="_blank">Nielsen's book</a>. It is an excellent read.</li>
<li> The textbook of <a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>, chapters 3 and 7 are the most relevant ones for the analysis here.</li>
<li> <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et al, arXiv 1803.08823</a>, <em>A high-bias, low-variance introduction to Machine Learning for physicists</em>, ArXiv:1803.08823.</li>
</ol>
<li> <a href="https://www.deeplearningbook.org/" target="_blank">Goodfellow, Bengio and Courville</a>, <em>Deep Learning</em>.
doc/Projects/2023/Project2/ipynb/Project2.ipynb
+22 -23
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "d183d07b",
"id": "6afd3fdb",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "a8e700e3",
"id": "21a10baa",
"metadata": {
"editable": true
},
@@ -22,14 +22,14 @@
"# Project 2 on Machine Learning, deadline November 13 (Midnight)\n",
"**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, Department of Physics, University of Oslo, Norway\n",
"\n",
"Date: **Oct 8, 2023**\n",
"Date: **Oct 9, 2023**\n",
"\n",
"Copyright 1999-2023, [Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html). Released under CC Attribution-NonCommercial 4.0 license"
]
},
{
"cell_type": "markdown",
"id": "dfe130a6",
"id": "4f67ce6e",
"metadata": {
"editable": true
},
@@ -75,7 +75,7 @@
},
{
"cell_type": "markdown",
"id": "0bb1ff8e",
"id": "67c6916e",
"metadata": {
"editable": true
},
@@ -92,18 +92,19 @@
"function.\n",
"\n",
"You should include in your analysis of the GD and SGD codes the following elements\n",
"1. A plain gradient descent with a fixed learning rate (you will need to tune it).\n",
"1. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.\n",
"\n",
"2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).\n",
"2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.\n",
"\n",
"3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n",
"3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.\n",
"\n",
"4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.\n",
"\n",
"5. Add RMSprop and Adam to your library of methods for tuning the learning rate.\n",
"\n",
"The lecture notes from [week 39 contain more\n",
"The lecture notes from [weeks 39 and 40contain more\n",
"details](https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html) and code examples. Feel free to use these examples.\n",
"1. Replace thereafter your analytical gradient with either **Autograd** or **JAX**\n",
"\n",
"In summary, you should \n",
"perform an analysis of the results for OLS and Ridge regression as\n",
@@ -126,7 +127,7 @@
},
{
"cell_type": "markdown",
"id": "18e8ecd4",
"id": "fff54502",
"metadata": {
"editable": true
},
@@ -165,7 +166,7 @@
},
{
"cell_type": "markdown",
"id": "1a264fae",
"id": "6e0f9012",
"metadata": {
"editable": true
},
@@ -177,7 +178,7 @@
},
{
"cell_type": "markdown",
"id": "d824e146",
"id": "075046bf",
"metadata": {
"editable": true
},
@@ -203,7 +204,7 @@
},
{
"cell_type": "markdown",
"id": "c74da15a",
"id": "080dadf2",
"metadata": {
"editable": true
},
@@ -215,7 +216,7 @@
},
{
"cell_type": "markdown",
"id": "8f765f8e",
"id": "e3418dd4",
"metadata": {
"editable": true
},
@@ -235,7 +236,7 @@
},
{
"cell_type": "markdown",
"id": "6d6372e7",
"id": "828ad79f",
"metadata": {
"editable": true
},
@@ -257,7 +258,7 @@
},
{
"cell_type": "markdown",
"id": "c62f38f2",
"id": "07059668",
"metadata": {
"editable": true
},
@@ -273,7 +274,7 @@
},
{
"cell_type": "markdown",
"id": "b345a171",
"id": "e88200db",
"metadata": {
"editable": true
},
@@ -282,16 +283,14 @@
"\n",
"1. The text of Michael Nielsen is highly recommended, see [Nielsen's book](http://neuralnetworksanddeeplearning.com/). It is an excellent read.\n",
"\n",
"2. The textbook of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570), chapters 3 and 7 are the most relevant ones for the analysis here. \n",
"2. [Mehta et al, arXiv 1803.08823](https://arxiv.org/abs/1803.08823), *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823.\n",
"\n",
"3. [Mehta et al, arXiv 1803.08823](https://arxiv.org/abs/1803.08823), *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823.\n",
"\n",
"d. [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/), *Deep Learning*."
"c. [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/), *Deep Learning*."
]
},
{
"cell_type": "markdown",
"id": "4d4f2980",
"id": "66733be9",
"metadata": {
"editable": true
},
@@ -322,7 +321,7 @@
},
{
"cell_type": "markdown",
"id": "fc20dc05",
"id": "5c49a400",
"metadata": {
"editable": true
},
doc/Projects/2023/Project2/ipynb/ipynb-Project2-src.tar.gz
BIN
View File
Binary file not shown.
doc/Projects/2023/Project2/pdf/Project2.p.tex
+9 -7
View File
@@ -149,7 +149,7 @@ Project 2 on Machine Learning, deadline November 13 (Midnight)
% --- begin date ---
\begin{center}
Oct 8, 2023
Oct 9, 2023
\end{center}
% --- end date ---
@@ -213,11 +213,11 @@ function.
You should include in your analysis of the GD and SGD codes the following elements
\begin{enumerate}
\item A plain gradient descent with a fixed learning rate (you will need to tune it).
\item A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.
\item Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
\item Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.
\item Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
\item Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.
\item Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
@@ -225,9 +225,13 @@ You should include in your analysis of the GD and SGD codes the following elemen
\end{enumerate}
\noindent
The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html}}{week 39 contain more
The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html}}{weeks 39 and 40contain more
details} and code examples. Feel free to use these examples.
\begin{enumerate}
\item Replace thereafter your analytical gradient with either \textbf{Autograd} or \textbf{JAX}
\end{enumerate}
\noindent
In summary, you should
perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
@@ -337,8 +341,6 @@ your final project 3, but now applied to other data sets.
\begin{enumerate}
\item The text of Michael Nielsen is highly recommended, see \href{{http://neuralnetworksanddeeplearning.com/}}{Nielsen's book}. It is an excellent read.
\item The textbook of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}, chapters 3 and 7 are the most relevant ones for the analysis here.
\item \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al, arXiv 1803.08823}, \emph{A high-bias, low-variance introduction to Machine Learning for physicists}, ArXiv:1803.08823.
\end{enumerate}
doc/Projects/2023/Project2/pdf/Project2.pdf
BIN
View File
Binary file not shown.
doc/Projects/2023/Project2/pdf/Project2.tex
+9 -7
View File
@@ -123,7 +123,7 @@ Project 2 on Machine Learning, deadline November 13 (Midnight)
% --- begin date ---
\begin{center}
Oct 8, 2023
Oct 9, 2023
\end{center}
% --- end date ---
@@ -187,11 +187,11 @@ function.
You should include in your analysis of the GD and SGD codes the following elements
\begin{enumerate}
\item A plain gradient descent with a fixed learning rate (you will need to tune it).
\item A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.
\item Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
\item Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.
\item Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
\item Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.
\item Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
@@ -199,9 +199,13 @@ You should include in your analysis of the GD and SGD codes the following elemen
\end{enumerate}
\noindent
The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html}}{week 39 contain more
The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html}}{weeks 39 and 40contain more
details} and code examples. Feel free to use these examples.
\begin{enumerate}
\item Replace thereafter your analytical gradient with either \textbf{Autograd} or \textbf{JAX}
\end{enumerate}
\noindent
In summary, you should
perform an analysis of the results for OLS and Ridge regression as
function of the chosen learning rates, the number of mini-batches and
@@ -311,8 +315,6 @@ your final project 3, but now applied to other data sets.
\begin{enumerate}
\item The text of Michael Nielsen is highly recommended, see \href{{http://neuralnetworksanddeeplearning.com/}}{Nielsen's book}. It is an excellent read.
\item The textbook of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}, chapters 3 and 7 are the most relevant ones for the analysis here.
\item \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al, arXiv 1803.08823}, \emph{A high-bias, low-variance introduction to Machine Learning for physicists}, ArXiv:1803.08823.
\end{enumerate}
doc/src/Projects/2023/Project2/Project2.do.txt
+5 -6
View File
@@ -54,13 +54,14 @@ function.
You should include in your analysis of the GD and SGD codes the following elements
o A plain gradient descent with a fixed learning rate (you will need to tune it).
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
o A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.
o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
o Add RMSprop and Adam to your library of methods for tuning the learning rate.
The lecture notes from "week 39 contain more
The lecture notes from "weeks 39 and 40contain more
details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples.
o Replace thereafter your analytical gradient with either _Autograd_ or _JAX_
In summary, you should
perform an analysis of the results for OLS and Ridge regression as
@@ -192,8 +193,6 @@ your final project 3, but now applied to other data sets.
o The text of Michael Nielsen is highly recommended, see "Nielsen's book":"http://neuralnetworksanddeeplearning.com/". It is an excellent read.
o The textbook of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer":"https://www.springer.com/gp/book/9780387848570", chapters 3 and 7 are the most relevant ones for the analysis here.
o "Mehta et al, arXiv 1803.08823":"https://arxiv.org/abs/1803.08823", *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823.
o "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/", *Deep Learning*.