lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update of exercises week 43

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-10-25 09:40:54 +02:00
parent ba5f9ea365
commit 0abaf08d94
13 changed files with 1484 additions and 1188 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/exercisesweek43.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/week43.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/_sources/exercisesweek43.ipynb
+286 -152
View File
@@ -2,8 +2,10 @@
"cells": [
{
"cell_type": "markdown",
"id": "ae832182",
"metadata": {},
"id": "d5ebb4c0",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek43.do.txt -->\n",
@@ -12,8 +14,10 @@
},
{
"cell_type": "markdown",
"id": "77f844d9",
"metadata": {},
"id": "812b4e46",
"metadata": {
"editable": true
},
"source": [
"# Exercises weeks 43 and 44 \n",
"**October 23-27, 2023**\n",
@@ -25,8 +29,10 @@
},
{
"cell_type": "markdown",
"id": "d5b983e3",
"metadata": {},
"id": "3230cd2f",
"metadata": {
"editable": true
},
"source": [
"# Overarching aims of the exercises weeks 43 and 44\n",
"\n",
@@ -63,8 +69,10 @@
},
{
"cell_type": "markdown",
"id": "7f6cc12f",
"metadata": {},
"id": "e3617d4e",
"metadata": {
"editable": true
},
"source": [
"## The AND and XOR Gates\n",
"\n",
@@ -99,8 +107,10 @@
},
{
"cell_type": "markdown",
"id": "c0b07e46",
"metadata": {},
"id": "54e1e7fc",
"metadata": {
"editable": true
},
"source": [
"## Representing the Data Sets\n",
"\n",
@@ -109,8 +119,10 @@
},
{
"cell_type": "markdown",
"id": "e0a9840a",
"metadata": {},
"id": "6b3c15cb",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -122,8 +134,10 @@
},
{
"cell_type": "markdown",
"id": "ff95d379",
"metadata": {},
"id": "acc25271",
"metadata": {
"editable": true
},
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
"\n",
@@ -150,8 +164,10 @@
},
{
"cell_type": "markdown",
"id": "36fa3466",
"metadata": {},
"id": "ffe0a840",
"metadata": {
"editable": true
},
"source": [
"## Setting up dimensionalities by hand\n",
"\n",
@@ -160,8 +176,10 @@
},
{
"cell_type": "markdown",
"id": "e2e5e808",
"metadata": {},
"id": "46abf545",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n",
@@ -171,16 +189,20 @@
},
{
"cell_type": "markdown",
"id": "419e3904",
"metadata": {},
"id": "86e105cc",
"metadata": {
"editable": true
},
"source": [
"Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives"
]
},
{
"cell_type": "markdown",
"id": "88a9712a",
"metadata": {},
"id": "5e1d21f1",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -192,16 +214,20 @@
},
{
"cell_type": "markdown",
"id": "036eee14",
"metadata": {},
"id": "692c6cbf",
"metadata": {
"editable": true
},
"source": [
"Assume also that the bias vector for the hidden layer is"
]
},
{
"cell_type": "markdown",
"id": "3c771918",
"metadata": {},
"id": "5d81e641",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n",
@@ -211,16 +237,20 @@
},
{
"cell_type": "markdown",
"id": "f5a4150d",
"metadata": {},
"id": "a42bdee4",
"metadata": {
"editable": true
},
"source": [
"Adding it gives us the input to the activation function of the hidden layer"
]
},
{
"cell_type": "markdown",
"id": "e4eb7753",
"metadata": {},
"id": "5116b854",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n",
@@ -232,16 +262,20 @@
},
{
"cell_type": "markdown",
"id": "b5442b32",
"metadata": {},
"id": "8dd26d09",
"metadata": {
"editable": true
},
"source": [
"Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have"
]
},
{
"cell_type": "markdown",
"id": "64d99f85",
"metadata": {},
"id": "ebf97c03",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -253,16 +287,20 @@
},
{
"cell_type": "markdown",
"id": "2aee4b6b",
"metadata": {},
"id": "262a4a3a",
"metadata": {
"editable": true
},
"source": [
"Assume also that the bias of the output layer is zero and that the weights of the output layer are"
]
},
{
"cell_type": "markdown",
"id": "884541eb",
"metadata": {},
"id": "f03ad8e7",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n",
@@ -272,37 +310,46 @@
},
{
"cell_type": "markdown",
"id": "3999a4d5",
"metadata": {},
"id": "36fe00c0",
"metadata": {
"editable": true
},
"source": [
"and multiplying with $\\boldsymbol{a}_h$ gives the output"
]
},
{
"cell_type": "markdown",
"id": "a4896b4d",
"metadata": {},
"id": "80ab43ac",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{a}_o=\\boldsymbol{w}_h^T\\begin{bmatrix} 0 & 0 \\\\\n",
"\\boldsymbol{a}_o=\\begin{bmatrix} 0 & 0 \\\\\n",
" 1 & 0 \\\\\n",
"\t\t 1 & 0 \\\\\n",
"\t\t 2 & 1 \\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n",
"\t\t 2 & 1 \\end{bmatrix}\\begin{bmatrix} 1 \\\\\n",
" -2\\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "2c74e7c0",
"metadata": {},
"id": "86bcfe49",
"metadata": {
"editable": true
},
"source": [
"the wanted result."
"the wanted result. Pay attention to the dimensionalities as well."
]
},
{
"cell_type": "markdown",
"id": "249b9804",
"metadata": {},
"id": "f19a899e",
"metadata": {
"editable": true
},
"source": [
"## Setting up the Neural Network\n",
"\n",
@@ -312,8 +359,11 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "afdd3100",
"metadata": {},
"id": "901ddca7",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -379,16 +429,20 @@
},
{
"cell_type": "markdown",
"id": "fb383164",
"metadata": {},
"id": "53f22266",
"metadata": {
"editable": true
},
"source": [
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
]
},
{
"cell_type": "markdown",
"id": "0d26394c",
"metadata": {},
"id": "5f665ac6",
"metadata": {
"editable": true
},
"source": [
"## The Code using Scikit-Learn"
]
@@ -396,8 +450,11 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "befc368c",
"metadata": {},
"id": "cb396cda",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -458,8 +515,10 @@
},
{
"cell_type": "markdown",
"id": "03895a9f",
"metadata": {},
"id": "c5978471",
"metadata": {
"editable": true
},
"source": [
"## Building a neural network code\n",
"\n",
@@ -475,8 +534,10 @@
},
{
"cell_type": "markdown",
"id": "6226915b",
"metadata": {},
"id": "7b82dc46",
"metadata": {
"editable": true
},
"source": [
"### Learning rate methods\n",
"\n",
@@ -495,8 +556,11 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "b6d5ea73",
"metadata": {},
"id": "de4dedfb",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -633,8 +697,10 @@
},
{
"cell_type": "markdown",
"id": "edb42bef",
"metadata": {},
"id": "bb621fce",
"metadata": {
"editable": true
},
"source": [
"### Usage of the above learning rate schedulers\n",
"\n",
@@ -647,8 +713,11 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "108c1209",
"metadata": {},
"id": "34ddb829",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
@@ -657,8 +726,10 @@
},
{
"cell_type": "markdown",
"id": "d1c423f0",
"metadata": {},
"id": "e43498d4",
"metadata": {
"editable": true
},
"source": [
"Here is a small example for how a segment of code using schedulers\n",
"could look. Switching out the schedulers is simple."
@@ -667,8 +738,11 @@
{
"cell_type": "code",
"execution_count": 5,
"id": "63c8c749",
"metadata": {},
"id": "f05b9625",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"weights = np.ones((3,3))\n",
@@ -686,8 +760,10 @@
},
{
"cell_type": "markdown",
"id": "7cb80992",
"metadata": {},
"id": "19cf9841",
"metadata": {
"editable": true
},
"source": [
"### Cost functions\n",
"\n",
@@ -700,8 +776,11 @@
{
"cell_type": "code",
"execution_count": 6,
"id": "a181dacb",
"metadata": {},
"id": "993efa8d",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -735,8 +814,10 @@
},
{
"cell_type": "markdown",
"id": "26a7fe37",
"metadata": {},
"id": "011c734c",
"metadata": {
"editable": true
},
"source": [
"Below we give a short example of how these cost function may be used\n",
"to obtain results if you wish to test them out on your own using\n",
@@ -746,8 +827,11 @@
{
"cell_type": "code",
"execution_count": 7,
"id": "5894bd31",
"metadata": {},
"id": "9e1b97f5",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from autograd import grad\n",
@@ -764,8 +848,10 @@
},
{
"cell_type": "markdown",
"id": "71141fab",
"metadata": {},
"id": "4acd87b2",
"metadata": {
"editable": true
},
"source": [
"### Activation functions\n",
"\n",
@@ -778,8 +864,11 @@
{
"cell_type": "code",
"execution_count": 8,
"id": "433e0886",
"metadata": {},
"id": "befa86ae",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -833,8 +922,10 @@
},
{
"cell_type": "markdown",
"id": "14c9d538",
"metadata": {},
"id": "c20aa75e",
"metadata": {
"editable": true
},
"source": [
"Below follows a short demonstration of how to use an activation\n",
"function. The derivative of the activation function will be important\n",
@@ -846,8 +937,11 @@
{
"cell_type": "code",
"execution_count": 9,
"id": "4c3a44ef",
"metadata": {},
"id": "e209f9e5",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"z = np.array([[4, 5, 6]]).T\n",
@@ -864,8 +958,10 @@
},
{
"cell_type": "markdown",
"id": "aa7f604e",
"metadata": {},
"id": "524b409b",
"metadata": {
"editable": true
},
"source": [
"### The Neural Network\n",
"\n",
@@ -886,8 +982,11 @@
{
"cell_type": "code",
"execution_count": 10,
"id": "585a227b",
"metadata": {},
"id": "083116d3",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import math\n",
@@ -1355,8 +1454,10 @@
},
{
"cell_type": "markdown",
"id": "7de704af",
"metadata": {},
"id": "e4ba55d0",
"metadata": {
"editable": true
},
"source": [
"Before we make a model, we will quickly generate a dataset we can use\n",
"for our linear regression problem as shown below"
@@ -1365,8 +1466,11 @@
{
"cell_type": "code",
"execution_count": 11,
"id": "212b0a35",
"metadata": {},
"id": "c72a22c6",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1406,8 +1510,10 @@
},
{
"cell_type": "markdown",
"id": "1ff4ec7b",
"metadata": {},
"id": "25b63b47",
"metadata": {
"editable": true
},
"source": [
"Now that we have our dataset ready for the regression, we can create\n",
"our regressor. Note that with the seed parameter, we can make sure our\n",
@@ -1420,8 +1526,11 @@
{
"cell_type": "code",
"execution_count": 12,
"id": "57d8eca3",
"metadata": {},
"id": "b6b4e461",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1432,8 +1541,10 @@
},
{
"cell_type": "markdown",
"id": "f1977bae",
"metadata": {},
"id": "d7860b74",
"metadata": {
"editable": true
},
"source": [
"We then fit our model with our training data using the scheduler of our choice."
]
@@ -1441,8 +1552,11 @@
{
"cell_type": "code",
"execution_count": 13,
"id": "31ae2a1e",
"metadata": {},
"id": "00522c73",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1453,8 +1567,10 @@
},
{
"cell_type": "markdown",
"id": "96a64da4",
"metadata": {},
"id": "a57bfb12",
"metadata": {
"editable": true
},
"source": [
"Due to the progress bar we can see the MSE (train_error) throughout\n",
"the FFNN's training. Note that the fit() function has some optional\n",
@@ -1467,8 +1583,11 @@
{
"cell_type": "code",
"execution_count": 14,
"id": "246499fc",
"metadata": {},
"id": "35259e41",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1478,8 +1597,10 @@
},
{
"cell_type": "markdown",
"id": "61d11e17",
"metadata": {},
"id": "4a403370",
"metadata": {
"editable": true
},
"source": [
"We see that given more epochs to train on, the regressor reaches a lower MSE.\n",
"\n",
@@ -1491,8 +1612,11 @@
{
"cell_type": "code",
"execution_count": 15,
"id": "b5b91899",
"metadata": {},
"id": "6c791130",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from sklearn.datasets import load_breast_cancer\n",
@@ -1514,8 +1638,11 @@
{
"cell_type": "code",
"execution_count": 16,
"id": "4bd01b3d",
"metadata": {},
"id": "0ac0258d",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1526,8 +1653,10 @@
},
{
"cell_type": "markdown",
"id": "3a138d66",
"metadata": {},
"id": "a9c74baa",
"metadata": {
"editable": true
},
"source": [
"We will now make use of our validation data by passing it into our fit function as a keyword argument"
]
@@ -1535,8 +1664,11 @@
{
"cell_type": "code",
"execution_count": 17,
"id": "faf1534e",
"metadata": {},
"id": "e5021bbf",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1547,8 +1679,10 @@
},
{
"cell_type": "markdown",
"id": "33d2d346",
"metadata": {},
"id": "ae0148bb",
"metadata": {
"editable": true
},
"source": [
"Finally, we will create a neural network with 2 hidden layers with activation functions."
]
@@ -1556,8 +1690,11 @@
{
"cell_type": "code",
"execution_count": 18,
"id": "a8f0a61c",
"metadata": {},
"id": "bb7e4340",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1573,8 +1710,11 @@
{
"cell_type": "code",
"execution_count": 19,
"id": "d651702d",
"metadata": {},
"id": "d0655afb",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1585,8 +1725,10 @@
},
{
"cell_type": "markdown",
"id": "200b64c0",
"metadata": {},
"id": "dc8a90af",
"metadata": {
"editable": true
},
"source": [
"### Multiclass classification\n",
"\n",
@@ -1598,8 +1740,11 @@
{
"cell_type": "code",
"execution_count": 20,
"id": "98b8d59b",
"metadata": {},
"id": "9305c08a",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from sklearn.datasets import load_digits\n",
@@ -1632,8 +1777,10 @@
},
{
"cell_type": "markdown",
"id": "cadd7e47",
"metadata": {},
"id": "8e208051",
"metadata": {
"editable": true
},
"source": [
"## Testing the XOR gate and other gates\n",
"\n",
@@ -1643,8 +1790,11 @@
{
"cell_type": "code",
"execution_count": 21,
"id": "a40e5cfe",
"metadata": {},
"id": "db016d41",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
@@ -1663,32 +1813,16 @@
},
{
"cell_type": "markdown",
"id": "b635db73",
"metadata": {},
"id": "d1384449",
"metadata": {
"editable": true
},
"source": [
"Not bad, but the results depend strongly on the learning reate. Try different learning rates."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
}
},
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
doc/LectureNotes/_build/html/_sources/week43.ipynb
+291 -290
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/exercisesweek43.html
+7 -5
View File
@@ -750,12 +750,13 @@ inputs <span class="math notranslate nohighlight">\(x_1\)</span> and <span class
<p>and multiplying with <span class="math notranslate nohighlight">\(\boldsymbol{a}_h\)</span> gives the output</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 &amp; 0 \\
\boldsymbol{a}_o=\begin{bmatrix} 0 &amp; 0 \\
1 &amp; 0 \\
1 &amp; 0 \\
2 &amp; 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
2 &amp; 1 \end{bmatrix}\begin{bmatrix} 1 \\
-2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
\end{split}\]</div>
<p>the wanted result.</p>
<p>the wanted result. Pay attention to the dimensionalities as well.</p>
</div>
<div class="section" id="setting-up-the-neural-network">
<h2>Setting up the Neural Network<a class="headerlink" href="#setting-up-the-neural-network" title="Permalink to this headline"></a></h2>
@@ -8456,8 +8457,9 @@ case.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Adam: Eta=0.0001, Lambda=0
[----------------------------------------] 0.000% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.4 | val_acc: 0.497
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> [----------------------------------------] 0.000% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.4 | val_acc: 0.497
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> [----------------------------------------] 0.1000% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.4 | val_acc: 0.497
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/html/week43.html
+8 -6
View File
@@ -1689,12 +1689,13 @@ inputs <span class="math notranslate nohighlight">\(x_1\)</span> and <span class
<p>and multiplying with <span class="math notranslate nohighlight">\(\boldsymbol{a}_h\)</span> gives the output</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 &amp; 0 \\
\boldsymbol{a}_o=\begin{bmatrix} 0 &amp; 0 \\
1 &amp; 0 \\
1 &amp; 0 \\
2 &amp; 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
2 &amp; 1 \end{bmatrix}\begin{bmatrix} 1 \\
-2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
\end{split}\]</div>
<p>the wanted result.</p>
<p>the wanted result. Pay attention to the dimensionalities as well.</p>
</div>
<div class="section" id="setting-up-the-neural-network">
<h2>Setting up the Neural Network<a class="headerlink" href="#setting-up-the-neural-network" title="Permalink to this headline"></a></h2>
@@ -6362,6 +6363,9 @@ case.</p>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Adam: Eta=0.001, Lambda=0
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> [----------------------------------------] 0.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392
</pre></div>
</div>
@@ -18823,9 +18827,7 @@ This is then passed through the activation:</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>probabilities = (n_inputs, n_categories) = (1437, 10)
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>probability that image 0 is in category 0,1,2,...,9 =
probability that image 0 is in category 0,1,2,...,9 =
[5.41511965e-04 2.17174962e-03 8.84355903e-03 1.44970586e-03
1.10378326e-04 5.08318298e-09 2.03256632e-04 1.92507116e-03
9.84443254e-01 3.11507992e-04]
doc/LectureNotes/_build/jupyter_execute/exercisesweek43.ipynb
+292 -139
View File
@@ -2,8 +2,10 @@
"cells": [
{
"cell_type": "markdown",
"id": "ae832182",
"metadata": {},
"id": "d5ebb4c0",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek43.do.txt -->\n",
@@ -12,8 +14,10 @@
},
{
"cell_type": "markdown",
"id": "77f844d9",
"metadata": {},
"id": "812b4e46",
"metadata": {
"editable": true
},
"source": [
"# Exercises weeks 43 and 44 \n",
"**October 23-27, 2023**\n",
@@ -25,8 +29,10 @@
},
{
"cell_type": "markdown",
"id": "d5b983e3",
"metadata": {},
"id": "3230cd2f",
"metadata": {
"editable": true
},
"source": [
"# Overarching aims of the exercises weeks 43 and 44\n",
"\n",
@@ -63,8 +69,10 @@
},
{
"cell_type": "markdown",
"id": "7f6cc12f",
"metadata": {},
"id": "e3617d4e",
"metadata": {
"editable": true
},
"source": [
"## The AND and XOR Gates\n",
"\n",
@@ -99,8 +107,10 @@
},
{
"cell_type": "markdown",
"id": "c0b07e46",
"metadata": {},
"id": "54e1e7fc",
"metadata": {
"editable": true
},
"source": [
"## Representing the Data Sets\n",
"\n",
@@ -109,8 +119,10 @@
},
{
"cell_type": "markdown",
"id": "e0a9840a",
"metadata": {},
"id": "6b3c15cb",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -122,8 +134,10 @@
},
{
"cell_type": "markdown",
"id": "ff95d379",
"metadata": {},
"id": "acc25271",
"metadata": {
"editable": true
},
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
"\n",
@@ -150,8 +164,10 @@
},
{
"cell_type": "markdown",
"id": "36fa3466",
"metadata": {},
"id": "ffe0a840",
"metadata": {
"editable": true
},
"source": [
"## Setting up dimensionalities by hand\n",
"\n",
@@ -160,8 +176,10 @@
},
{
"cell_type": "markdown",
"id": "e2e5e808",
"metadata": {},
"id": "46abf545",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n",
@@ -171,16 +189,20 @@
},
{
"cell_type": "markdown",
"id": "419e3904",
"metadata": {},
"id": "86e105cc",
"metadata": {
"editable": true
},
"source": [
"Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives"
]
},
{
"cell_type": "markdown",
"id": "88a9712a",
"metadata": {},
"id": "5e1d21f1",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -192,16 +214,20 @@
},
{
"cell_type": "markdown",
"id": "036eee14",
"metadata": {},
"id": "692c6cbf",
"metadata": {
"editable": true
},
"source": [
"Assume also that the bias vector for the hidden layer is"
]
},
{
"cell_type": "markdown",
"id": "3c771918",
"metadata": {},
"id": "5d81e641",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n",
@@ -211,16 +237,20 @@
},
{
"cell_type": "markdown",
"id": "f5a4150d",
"metadata": {},
"id": "a42bdee4",
"metadata": {
"editable": true
},
"source": [
"Adding it gives us the input to the activation function of the hidden layer"
]
},
{
"cell_type": "markdown",
"id": "e4eb7753",
"metadata": {},
"id": "5116b854",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n",
@@ -232,16 +262,20 @@
},
{
"cell_type": "markdown",
"id": "b5442b32",
"metadata": {},
"id": "8dd26d09",
"metadata": {
"editable": true
},
"source": [
"Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have"
]
},
{
"cell_type": "markdown",
"id": "64d99f85",
"metadata": {},
"id": "ebf97c03",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -253,16 +287,20 @@
},
{
"cell_type": "markdown",
"id": "2aee4b6b",
"metadata": {},
"id": "262a4a3a",
"metadata": {
"editable": true
},
"source": [
"Assume also that the bias of the output layer is zero and that the weights of the output layer are"
]
},
{
"cell_type": "markdown",
"id": "884541eb",
"metadata": {},
"id": "f03ad8e7",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n",
@@ -272,37 +310,46 @@
},
{
"cell_type": "markdown",
"id": "3999a4d5",
"metadata": {},
"id": "36fe00c0",
"metadata": {
"editable": true
},
"source": [
"and multiplying with $\\boldsymbol{a}_h$ gives the output"
]
},
{
"cell_type": "markdown",
"id": "a4896b4d",
"metadata": {},
"id": "80ab43ac",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{a}_o=\\boldsymbol{w}_h^T\\begin{bmatrix} 0 & 0 \\\\\n",
"\\boldsymbol{a}_o=\\begin{bmatrix} 0 & 0 \\\\\n",
" 1 & 0 \\\\\n",
"\t\t 1 & 0 \\\\\n",
"\t\t 2 & 1 \\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n",
"\t\t 2 & 1 \\end{bmatrix}\\begin{bmatrix} 1 \\\\\n",
" -2\\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "2c74e7c0",
"metadata": {},
"id": "86bcfe49",
"metadata": {
"editable": true
},
"source": [
"the wanted result."
"the wanted result. Pay attention to the dimensionalities as well."
]
},
{
"cell_type": "markdown",
"id": "249b9804",
"metadata": {},
"id": "f19a899e",
"metadata": {
"editable": true
},
"source": [
"## Setting up the Neural Network\n",
"\n",
@@ -312,8 +359,11 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "afdd3100",
"metadata": {},
"id": "901ddca7",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -390,16 +440,20 @@
},
{
"cell_type": "markdown",
"id": "fb383164",
"metadata": {},
"id": "53f22266",
"metadata": {
"editable": true
},
"source": [
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
]
},
{
"cell_type": "markdown",
"id": "0d26394c",
"metadata": {},
"id": "5f665ac6",
"metadata": {
"editable": true
},
"source": [
"## The Code using Scikit-Learn"
]
@@ -407,8 +461,11 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "befc368c",
"metadata": {},
"id": "cb396cda",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -712,8 +769,10 @@
},
{
"cell_type": "markdown",
"id": "03895a9f",
"metadata": {},
"id": "c5978471",
"metadata": {
"editable": true
},
"source": [
"## Building a neural network code\n",
"\n",
@@ -729,8 +788,10 @@
},
{
"cell_type": "markdown",
"id": "6226915b",
"metadata": {},
"id": "7b82dc46",
"metadata": {
"editable": true
},
"source": [
"### Learning rate methods\n",
"\n",
@@ -749,8 +810,11 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "b6d5ea73",
"metadata": {},
"id": "de4dedfb",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -887,8 +951,10 @@
},
{
"cell_type": "markdown",
"id": "edb42bef",
"metadata": {},
"id": "bb621fce",
"metadata": {
"editable": true
},
"source": [
"### Usage of the above learning rate schedulers\n",
"\n",
@@ -901,8 +967,11 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "108c1209",
"metadata": {},
"id": "34ddb829",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
@@ -911,8 +980,10 @@
},
{
"cell_type": "markdown",
"id": "d1c423f0",
"metadata": {},
"id": "e43498d4",
"metadata": {
"editable": true
},
"source": [
"Here is a small example for how a segment of code using schedulers\n",
"could look. Switching out the schedulers is simple."
@@ -921,8 +992,11 @@
{
"cell_type": "code",
"execution_count": 5,
"id": "63c8c749",
"metadata": {},
"id": "f05b9625",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -956,8 +1030,10 @@
},
{
"cell_type": "markdown",
"id": "7cb80992",
"metadata": {},
"id": "19cf9841",
"metadata": {
"editable": true
},
"source": [
"### Cost functions\n",
"\n",
@@ -970,8 +1046,11 @@
{
"cell_type": "code",
"execution_count": 6,
"id": "a181dacb",
"metadata": {},
"id": "993efa8d",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1005,8 +1084,10 @@
},
{
"cell_type": "markdown",
"id": "26a7fe37",
"metadata": {},
"id": "011c734c",
"metadata": {
"editable": true
},
"source": [
"Below we give a short example of how these cost function may be used\n",
"to obtain results if you wish to test them out on your own using\n",
@@ -1016,8 +1097,11 @@
{
"cell_type": "code",
"execution_count": 7,
"id": "5894bd31",
"metadata": {},
"id": "9e1b97f5",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -1045,8 +1129,10 @@
},
{
"cell_type": "markdown",
"id": "71141fab",
"metadata": {},
"id": "4acd87b2",
"metadata": {
"editable": true
},
"source": [
"### Activation functions\n",
"\n",
@@ -1059,8 +1145,11 @@
{
"cell_type": "code",
"execution_count": 8,
"id": "433e0886",
"metadata": {},
"id": "befa86ae",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1114,8 +1203,10 @@
},
{
"cell_type": "markdown",
"id": "14c9d538",
"metadata": {},
"id": "c20aa75e",
"metadata": {
"editable": true
},
"source": [
"Below follows a short demonstration of how to use an activation\n",
"function. The derivative of the activation function will be important\n",
@@ -1127,8 +1218,11 @@
{
"cell_type": "code",
"execution_count": 9,
"id": "4c3a44ef",
"metadata": {},
"id": "e209f9e5",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -1166,8 +1260,10 @@
},
{
"cell_type": "markdown",
"id": "aa7f604e",
"metadata": {},
"id": "524b409b",
"metadata": {
"editable": true
},
"source": [
"### The Neural Network\n",
"\n",
@@ -1188,8 +1284,11 @@
{
"cell_type": "code",
"execution_count": 10,
"id": "585a227b",
"metadata": {},
"id": "083116d3",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import math\n",
@@ -1657,8 +1756,10 @@
},
{
"cell_type": "markdown",
"id": "7de704af",
"metadata": {},
"id": "e4ba55d0",
"metadata": {
"editable": true
},
"source": [
"Before we make a model, we will quickly generate a dataset we can use\n",
"for our linear regression problem as shown below"
@@ -1667,8 +1768,11 @@
{
"cell_type": "code",
"execution_count": 11,
"id": "212b0a35",
"metadata": {},
"id": "c72a22c6",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1708,8 +1812,10 @@
},
{
"cell_type": "markdown",
"id": "1ff4ec7b",
"metadata": {},
"id": "25b63b47",
"metadata": {
"editable": true
},
"source": [
"Now that we have our dataset ready for the regression, we can create\n",
"our regressor. Note that with the seed parameter, we can make sure our\n",
@@ -1722,8 +1828,11 @@
{
"cell_type": "code",
"execution_count": 12,
"id": "57d8eca3",
"metadata": {},
"id": "b6b4e461",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1734,8 +1843,10 @@
},
{
"cell_type": "markdown",
"id": "f1977bae",
"metadata": {},
"id": "d7860b74",
"metadata": {
"editable": true
},
"source": [
"We then fit our model with our training data using the scheduler of our choice."
]
@@ -1743,8 +1854,11 @@
{
"cell_type": "code",
"execution_count": 13,
"id": "31ae2a1e",
"metadata": {},
"id": "00522c73",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -2573,8 +2687,10 @@
},
{
"cell_type": "markdown",
"id": "96a64da4",
"metadata": {},
"id": "a57bfb12",
"metadata": {
"editable": true
},
"source": [
"Due to the progress bar we can see the MSE (train_error) throughout\n",
"the FFNN's training. Note that the fit() function has some optional\n",
@@ -2587,8 +2703,11 @@
{
"cell_type": "code",
"execution_count": 14,
"id": "246499fc",
"metadata": {},
"id": "35259e41",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -10616,8 +10735,10 @@
},
{
"cell_type": "markdown",
"id": "61d11e17",
"metadata": {},
"id": "4a403370",
"metadata": {
"editable": true
},
"source": [
"We see that given more epochs to train on, the regressor reaches a lower MSE.\n",
"\n",
@@ -10629,8 +10750,11 @@
{
"cell_type": "code",
"execution_count": 15,
"id": "b5b91899",
"metadata": {},
"id": "6c791130",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from sklearn.datasets import load_breast_cancer\n",
@@ -10652,8 +10776,11 @@
{
"cell_type": "code",
"execution_count": 16,
"id": "4bd01b3d",
"metadata": {},
"id": "0ac0258d",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -10664,8 +10791,10 @@
},
{
"cell_type": "markdown",
"id": "3a138d66",
"metadata": {},
"id": "a9c74baa",
"metadata": {
"editable": true
},
"source": [
"We will now make use of our validation data by passing it into our fit function as a keyword argument"
]
@@ -10673,8 +10802,11 @@
{
"cell_type": "code",
"execution_count": 17,
"id": "faf1534e",
"metadata": {},
"id": "e5021bbf",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -18703,8 +18835,10 @@
},
{
"cell_type": "markdown",
"id": "33d2d346",
"metadata": {},
"id": "ae0148bb",
"metadata": {
"editable": true
},
"source": [
"Finally, we will create a neural network with 2 hidden layers with activation functions."
]
@@ -18712,8 +18846,11 @@
{
"cell_type": "code",
"execution_count": 18,
"id": "a8f0a61c",
"metadata": {},
"id": "bb7e4340",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -18729,14 +18866,23 @@
{
"cell_type": "code",
"execution_count": 19,
"id": "d651702d",
"metadata": {},
"id": "d0655afb",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Adam: Eta=0.0001, Lambda=0\n",
"Adam: Eta=0.0001, Lambda=0\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
" [----------------------------------------] 0.000% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.4 | val_acc: 0.497 "
]
@@ -26759,8 +26905,10 @@
},
{
"cell_type": "markdown",
"id": "200b64c0",
"metadata": {},
"id": "dc8a90af",
"metadata": {
"editable": true
},
"source": [
"### Multiclass classification\n",
"\n",
@@ -26772,8 +26920,11 @@
{
"cell_type": "code",
"execution_count": 20,
"id": "98b8d59b",
"metadata": {},
"id": "9305c08a",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -34830,8 +34981,10 @@
},
{
"cell_type": "markdown",
"id": "cadd7e47",
"metadata": {},
"id": "8e208051",
"metadata": {
"editable": true
},
"source": [
"## Testing the XOR gate and other gates\n",
"\n",
@@ -34841,8 +34994,11 @@
{
"cell_type": "code",
"execution_count": 21,
"id": "a40e5cfe",
"metadata": {},
"id": "db016d41",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"name": "stdout",
@@ -42879,19 +43035,16 @@
},
{
"cell_type": "markdown",
"id": "b635db73",
"metadata": {},
"id": "d1384449",
"metadata": {
"editable": true
},
"source": [
"Not bad, but the results depend strongly on the learning reate. Try different learning rates."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
doc/LectureNotes/_build/jupyter_execute/exercisesweek43.py
+4 -3
View File
@@ -160,13 +160,14 @@
# and multiplying with $\boldsymbol{a}_h$ gives the output
# $$
# \boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 & 0 \\
# \boldsymbol{a}_o=\begin{bmatrix} 0 & 0 \\
# 1 & 0 \\
# 1 & 0 \\
# 2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
# 2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\
# -2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
# $$
# the wanted result.
# the wanted result. Pay attention to the dimensionalities as well.
# ## Setting up the Neural Network
#
doc/LectureNotes/_build/jupyter_execute/week43.ipynb
+300 -299
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/jupyter_execute/week43.py
+4 -3
View File
@@ -210,13 +210,14 @@
# and multiplying with $\boldsymbol{a}_h$ gives the output
# $$
# \boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 & 0 \\
# \boldsymbol{a}_o=\begin{bmatrix} 0 & 0 \\
# 1 & 0 \\
# 1 & 0 \\
# 2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
# 2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\
# -2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
# $$
# the wanted result.
# the wanted result. Pay attention to the dimensionalities as well.
# ## Setting up the Neural Network
#
doc/LectureNotes/week43.ipynb
+291 -290
View File
File diff suppressed because it is too large Load Diff