diff --git a/doc/Projects/2023/Project2/html/._Project2-bs000.html b/doc/Projects/2023/Project2/html/._Project2-bs000.html index 7c501d743..f355757ab 100644 --- a/doc/Projects/2023/Project2/html/._Project2-bs000.html +++ b/doc/Projects/2023/Project2/html/._Project2-bs000.html @@ -145,7 +145,7 @@ MathJax.Hub.Config({
-

Oct 8, 2023

+

Oct 9, 2023


@@ -205,16 +205,18 @@ function.

You should include in your analysis of the GD and SGD codes the following elements

    -
  1. A plain gradient descent with a fixed learning rate (you will need to tune it).
    2. -
  2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
    3. -
  3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
    4. +
  4. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.
    5. +
  5. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.
    6. +
  6. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.
  7. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
  8. Add RMSprop and Adam to your library of methods for tuning the learning rate.
-

The lecture notes from week 39 contain more +

The lecture notes from weeks 39 and 40contain more details and code examples. Feel free to use these examples.

- +
    +
  1. Replace thereafter your analytical gradient with either Autograd or JAX
    2. +

In summary, you should perform an analysis of the results for OLS and Ridge regression as function of the chosen learning rates, the number of mini-batches and @@ -340,7 +342,6 @@ your final project 3, but now applied to other data sets.

  1. The text of Michael Nielsen is highly recommended, see Nielsen's book. It is an excellent read.
    2. -
  2. The textbook of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer, chapters 3 and 7 are the most relevant ones for the analysis here.
  3. Mehta et al, arXiv 1803.08823, A high-bias, low-variance introduction to Machine Learning for physicists, ArXiv:1803.08823.
  • Goodfellow, Bengio and Courville, Deep Learning. diff --git a/doc/Projects/2023/Project2/html/Project2-bs.html b/doc/Projects/2023/Project2/html/Project2-bs.html index 7c501d743..f355757ab 100644 --- a/doc/Projects/2023/Project2/html/Project2-bs.html +++ b/doc/Projects/2023/Project2/html/Project2-bs.html @@ -145,7 +145,7 @@ MathJax.Hub.Config({
    -

    Oct 8, 2023

    +

    Oct 9, 2023


    @@ -205,16 +205,18 @@ function.

    You should include in your analysis of the GD and SGD codes the following elements

      -
    1. A plain gradient descent with a fixed learning rate (you will need to tune it).
      2. -
    2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
      3. -
    3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
      4. +
    4. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.
      5. +
    5. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.
      6. +
    6. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.
    7. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
    8. Add RMSprop and Adam to your library of methods for tuning the learning rate.
    -

    The lecture notes from week 39 contain more +

    The lecture notes from weeks 39 and 40contain more details and code examples. Feel free to use these examples.

    - +
      +
    1. Replace thereafter your analytical gradient with either Autograd or JAX
      2. +

    In summary, you should perform an analysis of the results for OLS and Ridge regression as function of the chosen learning rates, the number of mini-batches and @@ -340,7 +342,6 @@ your final project 3, but now applied to other data sets.

    1. The text of Michael Nielsen is highly recommended, see Nielsen's book. It is an excellent read.
      2. -
    2. The textbook of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer, chapters 3 and 7 are the most relevant ones for the analysis here.
    3. Mehta et al, arXiv 1803.08823, A high-bias, low-variance introduction to Machine Learning for physicists, ArXiv:1803.08823.
  • Goodfellow, Bengio and Courville, Deep Learning. diff --git a/doc/Projects/2023/Project2/html/Project2.html b/doc/Projects/2023/Project2/html/Project2.html index fb63ebb8a..cb77a1749 100644 --- a/doc/Projects/2023/Project2/html/Project2.html +++ b/doc/Projects/2023/Project2/html/Project2.html @@ -187,7 +187,7 @@ MathJax.Hub.Config({
    -

    Oct 8, 2023

    +

    Oct 9, 2023


    Classification and Regression, from linear and logistic regression to neural networks

    @@ -244,16 +244,18 @@ function.

    You should include in your analysis of the GD and SGD codes the following elements

      -
    1. A plain gradient descent with a fixed learning rate (you will need to tune it).
      2. -
    2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).
      3. -
    3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
      4. +
    4. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.
      5. +
    5. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.
      6. +
    6. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.
    7. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.
    8. Add RMSprop and Adam to your library of methods for tuning the learning rate.
    -

    The lecture notes from week 39 contain more +

    The lecture notes from weeks 39 and 40contain more details and code examples. Feel free to use these examples.

    - +
      +
    1. Replace thereafter your analytical gradient with either Autograd or JAX
      2. +

    In summary, you should perform an analysis of the results for OLS and Ridge regression as function of the chosen learning rates, the number of mini-batches and @@ -379,7 +381,6 @@ your final project 3, but now applied to other data sets.

    1. The text of Michael Nielsen is highly recommended, see Nielsen's book. It is an excellent read.
      2. -
    2. The textbook of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer, chapters 3 and 7 are the most relevant ones for the analysis here.
    3. Mehta et al, arXiv 1803.08823, A high-bias, low-variance introduction to Machine Learning for physicists, ArXiv:1803.08823.
  • Goodfellow, Bengio and Courville, Deep Learning. diff --git a/doc/Projects/2023/Project2/ipynb/Project2.ipynb b/doc/Projects/2023/Project2/ipynb/Project2.ipynb index dc3477fd4..88c450dfd 100644 --- a/doc/Projects/2023/Project2/ipynb/Project2.ipynb +++ b/doc/Projects/2023/Project2/ipynb/Project2.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "d183d07b", + "id": "6afd3fdb", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "a8e700e3", + "id": "21a10baa", "metadata": { "editable": true }, @@ -22,14 +22,14 @@ "# Project 2 on Machine Learning, deadline November 13 (Midnight)\n", "**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, Department of Physics, University of Oslo, Norway\n", "\n", - "Date: **Oct 8, 2023**\n", + "Date: **Oct 9, 2023**\n", "\n", "Copyright 1999-2023, [Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html). Released under CC Attribution-NonCommercial 4.0 license" ] }, { "cell_type": "markdown", - "id": "dfe130a6", + "id": "4f67ce6e", "metadata": { "editable": true }, @@ -75,7 +75,7 @@ }, { "cell_type": "markdown", - "id": "0bb1ff8e", + "id": "67c6916e", "metadata": { "editable": true }, @@ -92,18 +92,19 @@ "function.\n", "\n", "You should include in your analysis of the GD and SGD codes the following elements\n", - "1. A plain gradient descent with a fixed learning rate (you will need to tune it).\n", + "1. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient.\n", "\n", - "2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate).\n", + "2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient.\n", "\n", - "3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n", + "3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient.\n", "\n", "4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.\n", "\n", "5. Add RMSprop and Adam to your library of methods for tuning the learning rate.\n", "\n", - "The lecture notes from [week 39 contain more\n", + "The lecture notes from [weeks 39 and 40contain more\n", "details](https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html) and code examples. Feel free to use these examples.\n", + "1. Replace thereafter your analytical gradient with either **Autograd** or **JAX**\n", "\n", "In summary, you should \n", "perform an analysis of the results for OLS and Ridge regression as\n", @@ -126,7 +127,7 @@ }, { "cell_type": "markdown", - "id": "18e8ecd4", + "id": "fff54502", "metadata": { "editable": true }, @@ -165,7 +166,7 @@ }, { "cell_type": "markdown", - "id": "1a264fae", + "id": "6e0f9012", "metadata": { "editable": true }, @@ -177,7 +178,7 @@ }, { "cell_type": "markdown", - "id": "d824e146", + "id": "075046bf", "metadata": { "editable": true }, @@ -203,7 +204,7 @@ }, { "cell_type": "markdown", - "id": "c74da15a", + "id": "080dadf2", "metadata": { "editable": true }, @@ -215,7 +216,7 @@ }, { "cell_type": "markdown", - "id": "8f765f8e", + "id": "e3418dd4", "metadata": { "editable": true }, @@ -235,7 +236,7 @@ }, { "cell_type": "markdown", - "id": "6d6372e7", + "id": "828ad79f", "metadata": { "editable": true }, @@ -257,7 +258,7 @@ }, { "cell_type": "markdown", - "id": "c62f38f2", + "id": "07059668", "metadata": { "editable": true }, @@ -273,7 +274,7 @@ }, { "cell_type": "markdown", - "id": "b345a171", + "id": "e88200db", "metadata": { "editable": true }, @@ -282,16 +283,14 @@ "\n", "1. The text of Michael Nielsen is highly recommended, see [Nielsen's book](http://neuralnetworksanddeeplearning.com/). It is an excellent read.\n", "\n", - "2. The textbook of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570), chapters 3 and 7 are the most relevant ones for the analysis here. \n", + "2. [Mehta et al, arXiv 1803.08823](https://arxiv.org/abs/1803.08823), *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823.\n", "\n", - "3. [Mehta et al, arXiv 1803.08823](https://arxiv.org/abs/1803.08823), *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823.\n", - "\n", - "d. [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/), *Deep Learning*." + "c. [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/), *Deep Learning*." ] }, { "cell_type": "markdown", - "id": "4d4f2980", + "id": "66733be9", "metadata": { "editable": true }, @@ -322,7 +321,7 @@ }, { "cell_type": "markdown", - "id": "fc20dc05", + "id": "5c49a400", "metadata": { "editable": true }, diff --git a/doc/Projects/2023/Project2/ipynb/ipynb-Project2-src.tar.gz b/doc/Projects/2023/Project2/ipynb/ipynb-Project2-src.tar.gz index 858e375ab..868a9e272 100644 Binary files a/doc/Projects/2023/Project2/ipynb/ipynb-Project2-src.tar.gz and b/doc/Projects/2023/Project2/ipynb/ipynb-Project2-src.tar.gz differ diff --git a/doc/Projects/2023/Project2/pdf/Project2.p.tex b/doc/Projects/2023/Project2/pdf/Project2.p.tex index cb2662e00..89ba20489 100644 --- a/doc/Projects/2023/Project2/pdf/Project2.p.tex +++ b/doc/Projects/2023/Project2/pdf/Project2.p.tex @@ -149,7 +149,7 @@ Project 2 on Machine Learning, deadline November 13 (Midnight) % --- begin date --- \begin{center} -Oct 8, 2023 +Oct 9, 2023 \end{center} % --- end date --- @@ -213,11 +213,11 @@ function. You should include in your analysis of the GD and SGD codes the following elements \begin{enumerate} -\item A plain gradient descent with a fixed learning rate (you will need to tune it). +\item A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient. -\item Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). +\item Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient. -\item Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc) +\item Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient. \item Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD. @@ -225,9 +225,13 @@ You should include in your analysis of the GD and SGD codes the following elemen \end{enumerate} \noindent -The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html}}{week 39 contain more +The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html}}{weeks 39 and 40contain more details} and code examples. Feel free to use these examples. +\begin{enumerate} +\item Replace thereafter your analytical gradient with either \textbf{Autograd} or \textbf{JAX} +\end{enumerate} +\noindent In summary, you should perform an analysis of the results for OLS and Ridge regression as function of the chosen learning rates, the number of mini-batches and @@ -337,8 +341,6 @@ your final project 3, but now applied to other data sets. \begin{enumerate} \item The text of Michael Nielsen is highly recommended, see \href{{http://neuralnetworksanddeeplearning.com/}}{Nielsen's book}. It is an excellent read. -\item The textbook of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}, chapters 3 and 7 are the most relevant ones for the analysis here. - \item \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al, arXiv 1803.08823}, \emph{A high-bias, low-variance introduction to Machine Learning for physicists}, ArXiv:1803.08823. \end{enumerate} diff --git a/doc/Projects/2023/Project2/pdf/Project2.pdf b/doc/Projects/2023/Project2/pdf/Project2.pdf index 4d9da4f8b..b68cff5c4 100644 Binary files a/doc/Projects/2023/Project2/pdf/Project2.pdf and b/doc/Projects/2023/Project2/pdf/Project2.pdf differ diff --git a/doc/Projects/2023/Project2/pdf/Project2.tex b/doc/Projects/2023/Project2/pdf/Project2.tex index 14cec4cfb..1ebe0982a 100644 --- a/doc/Projects/2023/Project2/pdf/Project2.tex +++ b/doc/Projects/2023/Project2/pdf/Project2.tex @@ -123,7 +123,7 @@ Project 2 on Machine Learning, deadline November 13 (Midnight) % --- begin date --- \begin{center} -Oct 8, 2023 +Oct 9, 2023 \end{center} % --- end date --- @@ -187,11 +187,11 @@ function. You should include in your analysis of the GD and SGD codes the following elements \begin{enumerate} -\item A plain gradient descent with a fixed learning rate (you will need to tune it). +\item A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient. -\item Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). +\item Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient. -\item Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc) +\item Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient. \item Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD. @@ -199,9 +199,13 @@ You should include in your analysis of the GD and SGD codes the following elemen \end{enumerate} \noindent -The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html}}{week 39 contain more +The lecture notes from \href{{https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html}}{weeks 39 and 40contain more details} and code examples. Feel free to use these examples. +\begin{enumerate} +\item Replace thereafter your analytical gradient with either \textbf{Autograd} or \textbf{JAX} +\end{enumerate} +\noindent In summary, you should perform an analysis of the results for OLS and Ridge regression as function of the chosen learning rates, the number of mini-batches and @@ -311,8 +315,6 @@ your final project 3, but now applied to other data sets. \begin{enumerate} \item The text of Michael Nielsen is highly recommended, see \href{{http://neuralnetworksanddeeplearning.com/}}{Nielsen's book}. It is an excellent read. -\item The textbook of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}, chapters 3 and 7 are the most relevant ones for the analysis here. - \item \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al, arXiv 1803.08823}, \emph{A high-bias, low-variance introduction to Machine Learning for physicists}, ArXiv:1803.08823. \end{enumerate} diff --git a/doc/src/Projects/2023/Project2/Project2.do.txt b/doc/src/Projects/2023/Project2/Project2.do.txt index 98b82d235..5ef1b3d4e 100644 --- a/doc/src/Projects/2023/Project2/Project2.do.txt +++ b/doc/src/Projects/2023/Project2/Project2.do.txt @@ -54,13 +54,14 @@ function. You should include in your analysis of the GD and SGD codes the following elements -o A plain gradient descent with a fixed learning rate (you will need to tune it). -o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). -o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc) +o A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression for the gradient. +o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Keep using the analytical expression for the gradient. +o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from weeks 39 and 40. Discuss the results as functions of the various parameters (size of batches, number of epochs etc). Use the analytical gradient. o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD. o Add RMSprop and Adam to your library of methods for tuning the learning rate. -The lecture notes from "week 39 contain more +The lecture notes from "weeks 39 and 40contain more details":"https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html" and code examples. Feel free to use these examples. +o Replace thereafter your analytical gradient with either _Autograd_ or _JAX_ In summary, you should perform an analysis of the results for OLS and Ridge regression as @@ -192,8 +193,6 @@ your final project 3, but now applied to other data sets. o The text of Michael Nielsen is highly recommended, see "Nielsen's book":"http://neuralnetworksanddeeplearning.com/". It is an excellent read. -o The textbook of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer":"https://www.springer.com/gp/book/9780387848570", chapters 3 and 7 are the most relevant ones for the analysis here. - o "Mehta et al, arXiv 1803.08823":"https://arxiv.org/abs/1803.08823", *A high-bias, low-variance introduction to Machine Learning for physicists*, ArXiv:1803.08823. o "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/", *Deep Learning*.