lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-10-08 17:04:26 +02:00
parent 87d1e4455c
commit dbbebedb06
114 changed files with 3736 additions and 1751 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/_sources/exercisesweek41.ipynb
+61 -139
View File
@@ -3,9 +3,7 @@
{
"cell_type": "markdown",
"id": "b18bcd06",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek41.do.txt -->\n",
@@ -15,9 +13,7 @@
{
"cell_type": "markdown",
"id": "7542d6aa",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"# Exercises week 41\n",
"**October 4-11, 2024**\n",
@@ -28,9 +24,7 @@
{
"cell_type": "markdown",
"id": "80943a15",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"# Overarching aims of the exercises this week\n",
"\n",
@@ -83,9 +77,7 @@
{
"cell_type": "markdown",
"id": "2095d197",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"# Code examples from week 39 and 40"
]
@@ -93,9 +85,7 @@
{
"cell_type": "markdown",
"id": "f428decb",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Code with a Number of Minibatches which varies, analytical gradient\n",
"\n",
@@ -106,10 +96,7 @@
"cell_type": "code",
"execution_count": 1,
"id": "ba38d454",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -185,9 +172,7 @@
{
"cell_type": "markdown",
"id": "de04b41a",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"In the above code, we have use replacement in setting up the\n",
"mini-batches. The discussion\n",
@@ -198,9 +183,7 @@
{
"cell_type": "markdown",
"id": "77fc1cca",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Momentum based GD\n",
"\n",
@@ -213,9 +196,7 @@
{
"cell_type": "markdown",
"id": "441d1f36",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n",
@@ -225,9 +206,7 @@
{
"cell_type": "markdown",
"id": "47434945",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"<!-- Equation labels as ordinary links -->\n",
"<div id=\"_auto1\"></div>\n",
@@ -243,9 +222,7 @@
{
"cell_type": "markdown",
"id": "f3ea5060",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"where we have introduced a momentum parameter $\\gamma$, with\n",
"$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n",
@@ -262,9 +239,7 @@
{
"cell_type": "markdown",
"id": "923628c8",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n",
@@ -274,9 +249,7 @@
{
"cell_type": "markdown",
"id": "5c94031c",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$."
]
@@ -284,9 +257,7 @@
{
"cell_type": "markdown",
"id": "f3f0e9c9",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Algorithms and codes for Adagrad, RMSprop and Adam\n",
"\n",
@@ -298,9 +269,7 @@
{
"cell_type": "markdown",
"id": "92253eff",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Practical tips\n",
"\n",
@@ -318,9 +287,7 @@
{
"cell_type": "markdown",
"id": "08209015",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Using Automatic differentation with OLS\n",
"\n",
@@ -333,10 +300,7 @@
"cell_type": "code",
"execution_count": 2,
"id": "f1f7d4aa",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# Using Autograd to calculate gradients for OLS\n",
@@ -393,9 +357,7 @@
{
"cell_type": "markdown",
"id": "1bc83f33",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Same code but now with momentum gradient descent"
]
@@ -404,10 +366,7 @@
"cell_type": "code",
"execution_count": 3,
"id": "dc2a3f65",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# Using Autograd to calculate gradients for OLS\n",
@@ -468,9 +427,7 @@
{
"cell_type": "markdown",
"id": "0ef007d0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## But noen of these can compete with Newton's method"
]
@@ -479,10 +436,7 @@
"cell_type": "code",
"execution_count": 4,
"id": "0e498aa4",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# Using Newton's method\n",
@@ -528,9 +482,7 @@
{
"cell_type": "markdown",
"id": "40292cf3",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Including Stochastic Gradient Descent with Autograd\n",
"In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**."
@@ -540,10 +492,7 @@
"cell_type": "code",
"execution_count": 5,
"id": "fa819b9d",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using SGD\n",
@@ -624,9 +573,7 @@
{
"cell_type": "markdown",
"id": "2ca466b4",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Same code but now with momentum gradient descent"
]
@@ -635,10 +582,7 @@
"cell_type": "code",
"execution_count": 6,
"id": "0d44a49c",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using SGD\n",
@@ -713,9 +657,7 @@
{
"cell_type": "markdown",
"id": "b82627f6",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## AdaGrad algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
"\n",
@@ -729,9 +671,7 @@
{
"cell_type": "markdown",
"id": "00d3aff0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Similar (second order function now) problem but now with AdaGrad"
]
@@ -740,10 +680,7 @@
"cell_type": "code",
"execution_count": 7,
"id": "6b85aacc",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n",
@@ -799,9 +736,7 @@
{
"cell_type": "markdown",
"id": "d8ddde38",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Running this code we note an almost perfect agreement with the results from matrix inversion."
]
@@ -809,9 +744,7 @@
{
"cell_type": "markdown",
"id": "ff15b503",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## RMSProp algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
"\n",
@@ -825,9 +758,7 @@
{
"cell_type": "markdown",
"id": "66f96d12",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## RMSprop for adaptive learning rate with Stochastic Gradient Descent"
]
@@ -836,10 +767,7 @@
"cell_type": "code",
"execution_count": 8,
"id": "888f1b4e",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n",
@@ -901,9 +829,7 @@
{
"cell_type": "markdown",
"id": "2e0860f7",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## ADAM algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
"\n",
@@ -917,9 +843,7 @@
{
"cell_type": "markdown",
"id": "ab4a9859",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## And finally [ADAM](https://arxiv.org/pdf/1412.6980.pdf)"
]
@@ -928,10 +852,7 @@
"cell_type": "code",
"execution_count": 9,
"id": "ccdd4d77",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n",
@@ -998,9 +919,7 @@
{
"cell_type": "markdown",
"id": "25ac988c",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n",
"\n",
@@ -1014,9 +933,7 @@
{
"cell_type": "markdown",
"id": "37d556d0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Getting started with Jax, note the way we import numpy"
]
@@ -1025,10 +942,7 @@
"cell_type": "code",
"execution_count": 10,
"id": "5b81d6e4",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import jax\n",
@@ -1042,9 +956,7 @@
{
"cell_type": "markdown",
"id": "c42db672",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### A warm-up example"
]
@@ -1053,10 +965,7 @@
"cell_type": "code",
"execution_count": 11,
"id": "98eb2f26",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"def function(x):\n",
@@ -1098,9 +1007,7 @@
{
"cell_type": "markdown",
"id": "8a5f19b5",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### A more advanced example"
]
@@ -1109,10 +1016,7 @@
"cell_type": "code",
"execution_count": 12,
"id": "d8f5eb38",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"backend = np\n",
@@ -1138,7 +1042,25 @@
]
}
],
"metadata": {},
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.18"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/LectureNotes/_build/html/_sources/project2.ipynb
+33 -43
View File
@@ -3,9 +3,7 @@
{
"cell_type": "markdown",
"id": "5b2f9dda",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html Project2.do.txt -->\n",
@@ -15,9 +13,7 @@
{
"cell_type": "markdown",
"id": "cacbd604",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"# Project 2 on Machine Learning, deadline November 4 (Midnight)\n",
"**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, Department of Physics, University of Oslo, Norway\n",
@@ -30,9 +26,7 @@
{
"cell_type": "markdown",
"id": "acb32119",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Classification and Regression, from linear and logistic regression to neural networks\n",
"\n",
@@ -76,9 +70,7 @@
{
"cell_type": "markdown",
"id": "027202f0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Part a): Write your own Stochastic Gradient Descent code, first step\n",
"\n",
@@ -132,9 +124,7 @@
{
"cell_type": "markdown",
"id": "9388fa74",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Part b): Writing your own Neural Network code\n",
"\n",
@@ -170,9 +160,7 @@
{
"cell_type": "markdown",
"id": "49666354",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Part c): Testing different activation functions\n",
"\n",
@@ -182,9 +170,7 @@
{
"cell_type": "markdown",
"id": "79aacf29",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Part d): Classification analysis using neural networks\n",
"\n",
@@ -208,9 +194,7 @@
{
"cell_type": "markdown",
"id": "42e22900",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\text{Accuracy} = \\frac{\\sum_{i=1}^n I(t_i = y_i)}{n} ,\n",
@@ -220,9 +204,7 @@
{
"cell_type": "markdown",
"id": "82ae763d",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$\n",
"otherwise if we have a binary classification problem. Here $t_i$\n",
@@ -240,9 +222,7 @@
{
"cell_type": "markdown",
"id": "1d6b84d1",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Part e): Write your Logistic Regression code, final step\n",
"\n",
@@ -262,9 +242,7 @@
{
"cell_type": "markdown",
"id": "0bce8832",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Part f) Critical evaluation of the various algorithms\n",
"\n",
@@ -278,9 +256,7 @@
{
"cell_type": "markdown",
"id": "51b1b29b",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Background literature\n",
"\n",
@@ -294,9 +270,7 @@
{
"cell_type": "markdown",
"id": "7e4ffbbd",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Introduction to numerical projects\n",
"\n",
@@ -325,9 +299,7 @@
{
"cell_type": "markdown",
"id": "56112b03",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Format for electronic delivery of report and programs\n",
"\n",
@@ -345,7 +317,25 @@
]
}
],
"metadata": {},
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.18"
}
},
"nbformat": 4,
"nbformat_minor": 5
}