lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-11-01 11:56:38 +01:00
parent 23145088bf
commit 1c1fde0f4c
7 changed files with 9652 additions and 1991 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/.ipynb_checkpoints/week41-checkpoint.ipynb
+3537
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/.ipynb_checkpoints/week44-checkpoint.ipynb
+5410
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/exercisesweek43.ipynb
+84 -217
View File
@@ -3,9 +3,7 @@
{
"cell_type": "markdown",
"id": "d5ebb4c0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek43.do.txt -->\n",
@@ -15,9 +13,7 @@
{
"cell_type": "markdown",
"id": "812b4e46",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"# Exercises weeks 43 and 44 \n",
"**October 23-27, 2023**\n",
@@ -30,9 +26,7 @@
{
"cell_type": "markdown",
"id": "3230cd2f",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"# Overarching aims of the exercises weeks 43 and 44\n",
"\n",
@@ -70,9 +64,7 @@
{
"cell_type": "markdown",
"id": "e3617d4e",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## The AND and XOR Gates\n",
"\n",
@@ -108,9 +100,7 @@
{
"cell_type": "markdown",
"id": "54e1e7fc",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Representing the Data Sets\n",
"\n",
@@ -120,9 +110,7 @@
{
"cell_type": "markdown",
"id": "6b3c15cb",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -135,9 +123,7 @@
{
"cell_type": "markdown",
"id": "acc25271",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
"\n",
@@ -165,9 +151,7 @@
{
"cell_type": "markdown",
"id": "ffe0a840",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Setting up dimensionalities by hand\n",
"\n",
@@ -177,9 +161,7 @@
{
"cell_type": "markdown",
"id": "46abf545",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n",
@@ -190,9 +172,7 @@
{
"cell_type": "markdown",
"id": "86e105cc",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives"
]
@@ -200,9 +180,7 @@
{
"cell_type": "markdown",
"id": "5e1d21f1",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -215,9 +193,7 @@
{
"cell_type": "markdown",
"id": "692c6cbf",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Assume also that the bias vector for the hidden layer is"
]
@@ -225,9 +201,7 @@
{
"cell_type": "markdown",
"id": "5d81e641",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n",
@@ -238,9 +212,7 @@
{
"cell_type": "markdown",
"id": "a42bdee4",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Adding it gives us the input to the activation function of the hidden layer"
]
@@ -248,9 +220,7 @@
{
"cell_type": "markdown",
"id": "5116b854",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n",
@@ -263,9 +233,7 @@
{
"cell_type": "markdown",
"id": "8dd26d09",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have"
]
@@ -273,9 +241,7 @@
{
"cell_type": "markdown",
"id": "ebf97c03",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -288,9 +254,7 @@
{
"cell_type": "markdown",
"id": "262a4a3a",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Assume also that the bias of the output layer is zero and that the weights of the output layer are"
]
@@ -298,9 +262,7 @@
{
"cell_type": "markdown",
"id": "f03ad8e7",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n",
@@ -311,9 +273,7 @@
{
"cell_type": "markdown",
"id": "36fe00c0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"and multiplying with $\\boldsymbol{a}_h$ gives the output"
]
@@ -321,9 +281,7 @@
{
"cell_type": "markdown",
"id": "80ab43ac",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{a}_o=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -337,9 +295,7 @@
{
"cell_type": "markdown",
"id": "86bcfe49",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"the wanted result. Pay attention to the dimensionalities as well."
]
@@ -347,9 +303,7 @@
{
"cell_type": "markdown",
"id": "f19a899e",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Setting up the Neural Network\n",
"\n",
@@ -360,10 +314,7 @@
"cell_type": "code",
"execution_count": 1,
"id": "901ddca7",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -430,9 +381,7 @@
{
"cell_type": "markdown",
"id": "53f22266",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
]
@@ -440,9 +389,7 @@
{
"cell_type": "markdown",
"id": "5f665ac6",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## The Code using Scikit-Learn"
]
@@ -451,10 +398,7 @@
"cell_type": "code",
"execution_count": 2,
"id": "cb396cda",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -516,9 +460,7 @@
{
"cell_type": "markdown",
"id": "c5978471",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Building a neural network code\n",
"\n",
@@ -535,9 +477,7 @@
{
"cell_type": "markdown",
"id": "7b82dc46",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Learning rate methods\n",
"\n",
@@ -557,10 +497,7 @@
"cell_type": "code",
"execution_count": 3,
"id": "de4dedfb",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -698,9 +635,7 @@
{
"cell_type": "markdown",
"id": "bb621fce",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Usage of the above learning rate schedulers\n",
"\n",
@@ -714,10 +649,7 @@
"cell_type": "code",
"execution_count": 4,
"id": "34ddb829",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
@@ -727,9 +659,7 @@
{
"cell_type": "markdown",
"id": "e43498d4",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Here is a small example for how a segment of code using schedulers\n",
"could look. Switching out the schedulers is simple."
@@ -739,10 +669,7 @@
"cell_type": "code",
"execution_count": 5,
"id": "f05b9625",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"weights = np.ones((3,3))\n",
@@ -761,9 +688,7 @@
{
"cell_type": "markdown",
"id": "19cf9841",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Cost functions\n",
"\n",
@@ -777,10 +702,7 @@
"cell_type": "code",
"execution_count": 6,
"id": "993efa8d",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -815,9 +737,7 @@
{
"cell_type": "markdown",
"id": "011c734c",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Below we give a short example of how these cost function may be used\n",
"to obtain results if you wish to test them out on your own using\n",
@@ -828,10 +748,7 @@
"cell_type": "code",
"execution_count": 7,
"id": "9e1b97f5",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"from autograd import grad\n",
@@ -849,9 +766,7 @@
{
"cell_type": "markdown",
"id": "4acd87b2",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Activation functions\n",
"\n",
@@ -865,10 +780,7 @@
"cell_type": "code",
"execution_count": 8,
"id": "befa86ae",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -923,9 +835,7 @@
{
"cell_type": "markdown",
"id": "c20aa75e",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Below follows a short demonstration of how to use an activation\n",
"function. The derivative of the activation function will be important\n",
@@ -938,10 +848,7 @@
"cell_type": "code",
"execution_count": 9,
"id": "e209f9e5",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"z = np.array([[4, 5, 6]]).T\n",
@@ -959,9 +866,7 @@
{
"cell_type": "markdown",
"id": "524b409b",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### The Neural Network\n",
"\n",
@@ -983,10 +888,7 @@
"cell_type": "code",
"execution_count": 10,
"id": "083116d3",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import math\n",
@@ -1455,9 +1357,7 @@
{
"cell_type": "markdown",
"id": "e4ba55d0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Before we make a model, we will quickly generate a dataset we can use\n",
"for our linear regression problem as shown below"
@@ -1467,10 +1367,7 @@
"cell_type": "code",
"execution_count": 11,
"id": "c72a22c6",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1511,9 +1408,7 @@
{
"cell_type": "markdown",
"id": "25b63b47",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Now that we have our dataset ready for the regression, we can create\n",
"our regressor. Note that with the seed parameter, we can make sure our\n",
@@ -1527,10 +1422,7 @@
"cell_type": "code",
"execution_count": 12,
"id": "b6b4e461",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1542,9 +1434,7 @@
{
"cell_type": "markdown",
"id": "d7860b74",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"We then fit our model with our training data using the scheduler of our choice."
]
@@ -1553,10 +1443,7 @@
"cell_type": "code",
"execution_count": 13,
"id": "00522c73",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1568,9 +1455,7 @@
{
"cell_type": "markdown",
"id": "a57bfb12",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Due to the progress bar we can see the MSE (train_error) throughout\n",
"the FFNN's training. Note that the fit() function has some optional\n",
@@ -1584,10 +1469,7 @@
"cell_type": "code",
"execution_count": 14,
"id": "35259e41",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1598,9 +1480,7 @@
{
"cell_type": "markdown",
"id": "4a403370",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"We see that given more epochs to train on, the regressor reaches a lower MSE.\n",
"\n",
@@ -1613,10 +1493,7 @@
"cell_type": "code",
"execution_count": 15,
"id": "6c791130",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_breast_cancer\n",
@@ -1639,10 +1516,7 @@
"cell_type": "code",
"execution_count": 16,
"id": "0ac0258d",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1654,9 +1528,7 @@
{
"cell_type": "markdown",
"id": "a9c74baa",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"We will now make use of our validation data by passing it into our fit function as a keyword argument"
]
@@ -1665,10 +1537,7 @@
"cell_type": "code",
"execution_count": 17,
"id": "e5021bbf",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1680,9 +1549,7 @@
{
"cell_type": "markdown",
"id": "ae0148bb",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Finally, we will create a neural network with 2 hidden layers with activation functions."
]
@@ -1691,10 +1558,7 @@
"cell_type": "code",
"execution_count": 18,
"id": "bb7e4340",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1711,10 +1575,7 @@
"cell_type": "code",
"execution_count": 19,
"id": "d0655afb",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1726,9 +1587,7 @@
{
"cell_type": "markdown",
"id": "dc8a90af",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Multiclass classification\n",
"\n",
@@ -1741,10 +1600,7 @@
"cell_type": "code",
"execution_count": 20,
"id": "9305c08a",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_digits\n",
@@ -1778,9 +1634,7 @@
{
"cell_type": "markdown",
"id": "8e208051",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Testing the XOR gate and other gates\n",
"\n",
@@ -1791,10 +1645,7 @@
"cell_type": "code",
"execution_count": 21,
"id": "db016d41",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
@@ -1814,15 +1665,31 @@
{
"cell_type": "markdown",
"id": "d1384449",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Not bad, but the results depend strongly on the learning reate. Try different learning rates."
]
}
],
"metadata": {},
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/LectureNotes/week41.ipynb
+231 -654
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/week42.ipynb
+141 -390
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/week44.ipynb
+249 -730
View File
File diff suppressed because it is too large Load Diff