lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-10-25 09:38:02 +02:00
parent b45fe1b743
commit ba5f9ea365
9 changed files with 602 additions and 462 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/exercisesweek43.ipynb
+286 -152
View File
@@ -2,8 +2,10 @@
"cells": [
{
"cell_type": "markdown",
"id": "ae832182",
"metadata": {},
"id": "d5ebb4c0",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek43.do.txt -->\n",
@@ -12,8 +14,10 @@
},
{
"cell_type": "markdown",
"id": "77f844d9",
"metadata": {},
"id": "812b4e46",
"metadata": {
"editable": true
},
"source": [
"# Exercises weeks 43 and 44 \n",
"**October 23-27, 2023**\n",
@@ -25,8 +29,10 @@
},
{
"cell_type": "markdown",
"id": "d5b983e3",
"metadata": {},
"id": "3230cd2f",
"metadata": {
"editable": true
},
"source": [
"# Overarching aims of the exercises weeks 43 and 44\n",
"\n",
@@ -63,8 +69,10 @@
},
{
"cell_type": "markdown",
"id": "7f6cc12f",
"metadata": {},
"id": "e3617d4e",
"metadata": {
"editable": true
},
"source": [
"## The AND and XOR Gates\n",
"\n",
@@ -99,8 +107,10 @@
},
{
"cell_type": "markdown",
"id": "c0b07e46",
"metadata": {},
"id": "54e1e7fc",
"metadata": {
"editable": true
},
"source": [
"## Representing the Data Sets\n",
"\n",
@@ -109,8 +119,10 @@
},
{
"cell_type": "markdown",
"id": "e0a9840a",
"metadata": {},
"id": "6b3c15cb",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -122,8 +134,10 @@
},
{
"cell_type": "markdown",
"id": "ff95d379",
"metadata": {},
"id": "acc25271",
"metadata": {
"editable": true
},
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
"\n",
@@ -150,8 +164,10 @@
},
{
"cell_type": "markdown",
"id": "36fa3466",
"metadata": {},
"id": "ffe0a840",
"metadata": {
"editable": true
},
"source": [
"## Setting up dimensionalities by hand\n",
"\n",
@@ -160,8 +176,10 @@
},
{
"cell_type": "markdown",
"id": "e2e5e808",
"metadata": {},
"id": "46abf545",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n",
@@ -171,16 +189,20 @@
},
{
"cell_type": "markdown",
"id": "419e3904",
"metadata": {},
"id": "86e105cc",
"metadata": {
"editable": true
},
"source": [
"Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives"
]
},
{
"cell_type": "markdown",
"id": "88a9712a",
"metadata": {},
"id": "5e1d21f1",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -192,16 +214,20 @@
},
{
"cell_type": "markdown",
"id": "036eee14",
"metadata": {},
"id": "692c6cbf",
"metadata": {
"editable": true
},
"source": [
"Assume also that the bias vector for the hidden layer is"
]
},
{
"cell_type": "markdown",
"id": "3c771918",
"metadata": {},
"id": "5d81e641",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n",
@@ -211,16 +237,20 @@
},
{
"cell_type": "markdown",
"id": "f5a4150d",
"metadata": {},
"id": "a42bdee4",
"metadata": {
"editable": true
},
"source": [
"Adding it gives us the input to the activation function of the hidden layer"
]
},
{
"cell_type": "markdown",
"id": "e4eb7753",
"metadata": {},
"id": "5116b854",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n",
@@ -232,16 +262,20 @@
},
{
"cell_type": "markdown",
"id": "b5442b32",
"metadata": {},
"id": "8dd26d09",
"metadata": {
"editable": true
},
"source": [
"Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have"
]
},
{
"cell_type": "markdown",
"id": "64d99f85",
"metadata": {},
"id": "ebf97c03",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -253,16 +287,20 @@
},
{
"cell_type": "markdown",
"id": "2aee4b6b",
"metadata": {},
"id": "262a4a3a",
"metadata": {
"editable": true
},
"source": [
"Assume also that the bias of the output layer is zero and that the weights of the output layer are"
]
},
{
"cell_type": "markdown",
"id": "884541eb",
"metadata": {},
"id": "f03ad8e7",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n",
@@ -272,37 +310,46 @@
},
{
"cell_type": "markdown",
"id": "3999a4d5",
"metadata": {},
"id": "36fe00c0",
"metadata": {
"editable": true
},
"source": [
"and multiplying with $\\boldsymbol{a}_h$ gives the output"
]
},
{
"cell_type": "markdown",
"id": "a4896b4d",
"metadata": {},
"id": "80ab43ac",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{a}_o=\\boldsymbol{w}_h^T\\begin{bmatrix} 0 & 0 \\\\\n",
"\\boldsymbol{a}_o=\\begin{bmatrix} 0 & 0 \\\\\n",
" 1 & 0 \\\\\n",
"\t\t 1 & 0 \\\\\n",
"\t\t 2 & 1 \\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n",
"\t\t 2 & 1 \\end{bmatrix}\\begin{bmatrix} 1 \\\\\n",
" -2\\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "2c74e7c0",
"metadata": {},
"id": "86bcfe49",
"metadata": {
"editable": true
},
"source": [
"the wanted result."
"the wanted result. Pay attention to the dimensionalities as well."
]
},
{
"cell_type": "markdown",
"id": "249b9804",
"metadata": {},
"id": "f19a899e",
"metadata": {
"editable": true
},
"source": [
"## Setting up the Neural Network\n",
"\n",
@@ -312,8 +359,11 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "afdd3100",
"metadata": {},
"id": "901ddca7",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -379,16 +429,20 @@
},
{
"cell_type": "markdown",
"id": "fb383164",
"metadata": {},
"id": "53f22266",
"metadata": {
"editable": true
},
"source": [
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
]
},
{
"cell_type": "markdown",
"id": "0d26394c",
"metadata": {},
"id": "5f665ac6",
"metadata": {
"editable": true
},
"source": [
"## The Code using Scikit-Learn"
]
@@ -396,8 +450,11 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "befc368c",
"metadata": {},
"id": "cb396cda",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -458,8 +515,10 @@
},
{
"cell_type": "markdown",
"id": "03895a9f",
"metadata": {},
"id": "c5978471",
"metadata": {
"editable": true
},
"source": [
"## Building a neural network code\n",
"\n",
@@ -475,8 +534,10 @@
},
{
"cell_type": "markdown",
"id": "6226915b",
"metadata": {},
"id": "7b82dc46",
"metadata": {
"editable": true
},
"source": [
"### Learning rate methods\n",
"\n",
@@ -495,8 +556,11 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "b6d5ea73",
"metadata": {},
"id": "de4dedfb",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -633,8 +697,10 @@
},
{
"cell_type": "markdown",
"id": "edb42bef",
"metadata": {},
"id": "bb621fce",
"metadata": {
"editable": true
},
"source": [
"### Usage of the above learning rate schedulers\n",
"\n",
@@ -647,8 +713,11 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "108c1209",
"metadata": {},
"id": "34ddb829",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
@@ -657,8 +726,10 @@
},
{
"cell_type": "markdown",
"id": "d1c423f0",
"metadata": {},
"id": "e43498d4",
"metadata": {
"editable": true
},
"source": [
"Here is a small example for how a segment of code using schedulers\n",
"could look. Switching out the schedulers is simple."
@@ -667,8 +738,11 @@
{
"cell_type": "code",
"execution_count": 5,
"id": "63c8c749",
"metadata": {},
"id": "f05b9625",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"weights = np.ones((3,3))\n",
@@ -686,8 +760,10 @@
},
{
"cell_type": "markdown",
"id": "7cb80992",
"metadata": {},
"id": "19cf9841",
"metadata": {
"editable": true
},
"source": [
"### Cost functions\n",
"\n",
@@ -700,8 +776,11 @@
{
"cell_type": "code",
"execution_count": 6,
"id": "a181dacb",
"metadata": {},
"id": "993efa8d",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -735,8 +814,10 @@
},
{
"cell_type": "markdown",
"id": "26a7fe37",
"metadata": {},
"id": "011c734c",
"metadata": {
"editable": true
},
"source": [
"Below we give a short example of how these cost function may be used\n",
"to obtain results if you wish to test them out on your own using\n",
@@ -746,8 +827,11 @@
{
"cell_type": "code",
"execution_count": 7,
"id": "5894bd31",
"metadata": {},
"id": "9e1b97f5",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from autograd import grad\n",
@@ -764,8 +848,10 @@
},
{
"cell_type": "markdown",
"id": "71141fab",
"metadata": {},
"id": "4acd87b2",
"metadata": {
"editable": true
},
"source": [
"### Activation functions\n",
"\n",
@@ -778,8 +864,11 @@
{
"cell_type": "code",
"execution_count": 8,
"id": "433e0886",
"metadata": {},
"id": "befa86ae",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -833,8 +922,10 @@
},
{
"cell_type": "markdown",
"id": "14c9d538",
"metadata": {},
"id": "c20aa75e",
"metadata": {
"editable": true
},
"source": [
"Below follows a short demonstration of how to use an activation\n",
"function. The derivative of the activation function will be important\n",
@@ -846,8 +937,11 @@
{
"cell_type": "code",
"execution_count": 9,
"id": "4c3a44ef",
"metadata": {},
"id": "e209f9e5",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"z = np.array([[4, 5, 6]]).T\n",
@@ -864,8 +958,10 @@
},
{
"cell_type": "markdown",
"id": "aa7f604e",
"metadata": {},
"id": "524b409b",
"metadata": {
"editable": true
},
"source": [
"### The Neural Network\n",
"\n",
@@ -886,8 +982,11 @@
{
"cell_type": "code",
"execution_count": 10,
"id": "585a227b",
"metadata": {},
"id": "083116d3",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import math\n",
@@ -1355,8 +1454,10 @@
},
{
"cell_type": "markdown",
"id": "7de704af",
"metadata": {},
"id": "e4ba55d0",
"metadata": {
"editable": true
},
"source": [
"Before we make a model, we will quickly generate a dataset we can use\n",
"for our linear regression problem as shown below"
@@ -1365,8 +1466,11 @@
{
"cell_type": "code",
"execution_count": 11,
"id": "212b0a35",
"metadata": {},
"id": "c72a22c6",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1406,8 +1510,10 @@
},
{
"cell_type": "markdown",
"id": "1ff4ec7b",
"metadata": {},
"id": "25b63b47",
"metadata": {
"editable": true
},
"source": [
"Now that we have our dataset ready for the regression, we can create\n",
"our regressor. Note that with the seed parameter, we can make sure our\n",
@@ -1420,8 +1526,11 @@
{
"cell_type": "code",
"execution_count": 12,
"id": "57d8eca3",
"metadata": {},
"id": "b6b4e461",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1432,8 +1541,10 @@
},
{
"cell_type": "markdown",
"id": "f1977bae",
"metadata": {},
"id": "d7860b74",
"metadata": {
"editable": true
},
"source": [
"We then fit our model with our training data using the scheduler of our choice."
]
@@ -1441,8 +1552,11 @@
{
"cell_type": "code",
"execution_count": 13,
"id": "31ae2a1e",
"metadata": {},
"id": "00522c73",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1453,8 +1567,10 @@
},
{
"cell_type": "markdown",
"id": "96a64da4",
"metadata": {},
"id": "a57bfb12",
"metadata": {
"editable": true
},
"source": [
"Due to the progress bar we can see the MSE (train_error) throughout\n",
"the FFNN's training. Note that the fit() function has some optional\n",
@@ -1467,8 +1583,11 @@
{
"cell_type": "code",
"execution_count": 14,
"id": "246499fc",
"metadata": {},
"id": "35259e41",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1478,8 +1597,10 @@
},
{
"cell_type": "markdown",
"id": "61d11e17",
"metadata": {},
"id": "4a403370",
"metadata": {
"editable": true
},
"source": [
"We see that given more epochs to train on, the regressor reaches a lower MSE.\n",
"\n",
@@ -1491,8 +1612,11 @@
{
"cell_type": "code",
"execution_count": 15,
"id": "b5b91899",
"metadata": {},
"id": "6c791130",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from sklearn.datasets import load_breast_cancer\n",
@@ -1514,8 +1638,11 @@
{
"cell_type": "code",
"execution_count": 16,
"id": "4bd01b3d",
"metadata": {},
"id": "0ac0258d",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1526,8 +1653,10 @@
},
{
"cell_type": "markdown",
"id": "3a138d66",
"metadata": {},
"id": "a9c74baa",
"metadata": {
"editable": true
},
"source": [
"We will now make use of our validation data by passing it into our fit function as a keyword argument"
]
@@ -1535,8 +1664,11 @@
{
"cell_type": "code",
"execution_count": 17,
"id": "faf1534e",
"metadata": {},
"id": "e5021bbf",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1547,8 +1679,10 @@
},
{
"cell_type": "markdown",
"id": "33d2d346",
"metadata": {},
"id": "ae0148bb",
"metadata": {
"editable": true
},
"source": [
"Finally, we will create a neural network with 2 hidden layers with activation functions."
]
@@ -1556,8 +1690,11 @@
{
"cell_type": "code",
"execution_count": 18,
"id": "a8f0a61c",
"metadata": {},
"id": "bb7e4340",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1573,8 +1710,11 @@
{
"cell_type": "code",
"execution_count": 19,
"id": "d651702d",
"metadata": {},
"id": "d0655afb",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1585,8 +1725,10 @@
},
{
"cell_type": "markdown",
"id": "200b64c0",
"metadata": {},
"id": "dc8a90af",
"metadata": {
"editable": true
},
"source": [
"### Multiclass classification\n",
"\n",
@@ -1598,8 +1740,11 @@
{
"cell_type": "code",
"execution_count": 20,
"id": "98b8d59b",
"metadata": {},
"id": "9305c08a",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from sklearn.datasets import load_digits\n",
@@ -1632,8 +1777,10 @@
},
{
"cell_type": "markdown",
"id": "cadd7e47",
"metadata": {},
"id": "8e208051",
"metadata": {
"editable": true
},
"source": [
"## Testing the XOR gate and other gates\n",
"\n",
@@ -1643,8 +1790,11 @@
{
"cell_type": "code",
"execution_count": 21,
"id": "a40e5cfe",
"metadata": {},
"id": "db016d41",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
@@ -1663,32 +1813,16 @@
},
{
"cell_type": "markdown",
"id": "b635db73",
"metadata": {},
"id": "d1384449",
"metadata": {
"editable": true
},
"source": [
"Not bad, but the results depend strongly on the learning reate. Try different learning rates."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
}
},
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
doc/pub/week43/html/week43-bs.html
+4 -3
View File
@@ -698,13 +698,14 @@ $$
<p>and multiplying with \( \boldsymbol{a}_h \) gives the output</p>
$$
\boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 & 0 \\
\boldsymbol{a}_o=\begin{bmatrix} 0 & 0 \\
1 & 0 \\
1 & 0 \\
2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\
-2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
$$
<p>the wanted result.</p>
<p>the wanted result. Pay attention to the dimensionalities as well.</p>
<!-- !split -->
<h2 id="setting-up-the-neural-network" class="anchor">Setting up the Neural Network </h2>
doc/pub/week43/html/week43-reveal.html
+4 -3
View File
@@ -426,14 +426,15 @@ $$
<p>and multiplying with \( \boldsymbol{a}_h \) gives the output</p>
<p>&nbsp;<br>
$$
\boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 & 0 \\
\boldsymbol{a}_o=\begin{bmatrix} 0 & 0 \\
1 & 0 \\
1 & 0 \\
2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\
-2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
$$
<p>&nbsp;<br>
<p>the wanted result.</p>
<p>the wanted result. Pay attention to the dimensionalities as well.</p>
</section>
<section>
doc/pub/week43/html/week43-solarized.html
+4 -3
View File
@@ -578,13 +578,14 @@ $$
<p>and multiplying with \( \boldsymbol{a}_h \) gives the output</p>
$$
\boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 & 0 \\
\boldsymbol{a}_o=\begin{bmatrix} 0 & 0 \\
1 & 0 \\
1 & 0 \\
2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\
-2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
$$
<p>the wanted result.</p>
<p>the wanted result. Pay attention to the dimensionalities as well.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-neural-network">Setting up the Neural Network </h2>
doc/pub/week43/html/week43.html
+4 -3
View File
@@ -655,13 +655,14 @@ $$
<p>and multiplying with \( \boldsymbol{a}_h \) gives the output</p>
$$
\boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 & 0 \\
\boldsymbol{a}_o=\begin{bmatrix} 0 & 0 \\
1 & 0 \\
1 & 0 \\
2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\
-2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
$$
<p>the wanted result.</p>
<p>the wanted result. Pay attention to the dimensionalities as well.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-neural-network">Setting up the Neural Network </h2>
doc/pub/week43/ipynb/ipynb-week43-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week43/ipynb/week43.ipynb
+291 -290
View File
File diff suppressed because it is too large Load Diff
doc/src/week43/exercisesweek43.do.txt
+4 -3
View File
@@ -134,12 +134,13 @@ Assume also that the bias of the output layer is zero and that the weights of th
!et
and multiplying with $\bm{a}_h$ gives the output
!bt
\bm{a}_o=\bm{w}_h^T\begin{bmatrix} 0 & 0 \\
\bm{a}_o=\begin{bmatrix} 0 & 0 \\
1 & 0 \\
1 & 0 \\
2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\
-2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
!et
the wanted result.
the wanted result. Pay attention to the dimensionalities as well.
!split
doc/src/week43/week43.do.txt
+5 -5
View File
@@ -61,6 +61,7 @@ cost function.
Their inputs and outputs can be
summarized using the following tables, first for the OR gate with
inputs $x_1$ and $x_2$ and outputs $y$:
@@ -173,12 +174,13 @@ Assume also that the bias of the output layer is zero and that the weights of th
!et
and multiplying with $\bm{a}_h$ gives the output
!bt
\bm{a}_o=\bm{w}_h^T\begin{bmatrix} 0 & 0 \\
\bm{a}_o=\begin{bmatrix} 0 & 0 \\
1 & 0 \\
1 & 0 \\
2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\
-2\end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
!et
the wanted result.
the wanted result. Pay attention to the dimensionalities as well.
!split
@@ -1321,8 +1323,6 @@ scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)
!ec
Not bad, but the results depend strongly on the learning reate. Try different learning rates.
!split
===== Lecture Thursday October 26 =====