lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on notes

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-11-06 06:36:30 +01:00
parent 218167a73f
commit 71b528ed83
74 changed files with 14208 additions and 5101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/_images/week44_135_2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 33 KiB

doc/LectureNotes/_build/html/_images/week44_135_3.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

doc/LectureNotes/_build/html/_images/week44_139_0.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 123 KiB

doc/LectureNotes/_build/html/_images/week44_149_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

doc/LectureNotes/_build/html/_images/week45_15_0.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 127 KiB

doc/LectureNotes/_build/html/_images/week45_15_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 261 KiB

doc/LectureNotes/_build/html/_images/week45_22_2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

doc/LectureNotes/_build/html/_images/week45_22_3.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

doc/LectureNotes/_build/html/_images/week45_22_4.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

doc/LectureNotes/_build/html/_images/week45_5_0.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

doc/LectureNotes/_build/html/_sources/exercisesweek43.ipynb
+84 -217
View File
@@ -3,9 +3,7 @@
{
"cell_type": "markdown",
"id": "d5ebb4c0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek43.do.txt -->\n",
@@ -15,9 +13,7 @@
{
"cell_type": "markdown",
"id": "812b4e46",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"# Exercises weeks 43 and 44 \n",
"**October 23-27, 2023**\n",
@@ -30,9 +26,7 @@
{
"cell_type": "markdown",
"id": "3230cd2f",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"# Overarching aims of the exercises weeks 43 and 44\n",
"\n",
@@ -70,9 +64,7 @@
{
"cell_type": "markdown",
"id": "e3617d4e",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## The AND and XOR Gates\n",
"\n",
@@ -108,9 +100,7 @@
{
"cell_type": "markdown",
"id": "54e1e7fc",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Representing the Data Sets\n",
"\n",
@@ -120,9 +110,7 @@
{
"cell_type": "markdown",
"id": "6b3c15cb",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -135,9 +123,7 @@
{
"cell_type": "markdown",
"id": "acc25271",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
"\n",
@@ -165,9 +151,7 @@
{
"cell_type": "markdown",
"id": "ffe0a840",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Setting up dimensionalities by hand\n",
"\n",
@@ -177,9 +161,7 @@
{
"cell_type": "markdown",
"id": "46abf545",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n",
@@ -190,9 +172,7 @@
{
"cell_type": "markdown",
"id": "86e105cc",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives"
]
@@ -200,9 +180,7 @@
{
"cell_type": "markdown",
"id": "5e1d21f1",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -215,9 +193,7 @@
{
"cell_type": "markdown",
"id": "692c6cbf",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Assume also that the bias vector for the hidden layer is"
]
@@ -225,9 +201,7 @@
{
"cell_type": "markdown",
"id": "5d81e641",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n",
@@ -238,9 +212,7 @@
{
"cell_type": "markdown",
"id": "a42bdee4",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Adding it gives us the input to the activation function of the hidden layer"
]
@@ -248,9 +220,7 @@
{
"cell_type": "markdown",
"id": "5116b854",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n",
@@ -263,9 +233,7 @@
{
"cell_type": "markdown",
"id": "8dd26d09",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have"
]
@@ -273,9 +241,7 @@
{
"cell_type": "markdown",
"id": "ebf97c03",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -288,9 +254,7 @@
{
"cell_type": "markdown",
"id": "262a4a3a",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Assume also that the bias of the output layer is zero and that the weights of the output layer are"
]
@@ -298,9 +262,7 @@
{
"cell_type": "markdown",
"id": "f03ad8e7",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n",
@@ -311,9 +273,7 @@
{
"cell_type": "markdown",
"id": "36fe00c0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"and multiplying with $\\boldsymbol{a}_h$ gives the output"
]
@@ -321,9 +281,7 @@
{
"cell_type": "markdown",
"id": "80ab43ac",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{a}_o=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -337,9 +295,7 @@
{
"cell_type": "markdown",
"id": "86bcfe49",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"the wanted result. Pay attention to the dimensionalities as well."
]
@@ -347,9 +303,7 @@
{
"cell_type": "markdown",
"id": "f19a899e",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Setting up the Neural Network\n",
"\n",
@@ -360,10 +314,7 @@
"cell_type": "code",
"execution_count": 1,
"id": "901ddca7",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -430,9 +381,7 @@
{
"cell_type": "markdown",
"id": "53f22266",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
]
@@ -440,9 +389,7 @@
{
"cell_type": "markdown",
"id": "5f665ac6",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## The Code using Scikit-Learn"
]
@@ -451,10 +398,7 @@
"cell_type": "code",
"execution_count": 2,
"id": "cb396cda",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -516,9 +460,7 @@
{
"cell_type": "markdown",
"id": "c5978471",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Building a neural network code\n",
"\n",
@@ -535,9 +477,7 @@
{
"cell_type": "markdown",
"id": "7b82dc46",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Learning rate methods\n",
"\n",
@@ -557,10 +497,7 @@
"cell_type": "code",
"execution_count": 3,
"id": "de4dedfb",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -698,9 +635,7 @@
{
"cell_type": "markdown",
"id": "bb621fce",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Usage of the above learning rate schedulers\n",
"\n",
@@ -714,10 +649,7 @@
"cell_type": "code",
"execution_count": 4,
"id": "34ddb829",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
@@ -727,9 +659,7 @@
{
"cell_type": "markdown",
"id": "e43498d4",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Here is a small example for how a segment of code using schedulers\n",
"could look. Switching out the schedulers is simple."
@@ -739,10 +669,7 @@
"cell_type": "code",
"execution_count": 5,
"id": "f05b9625",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"weights = np.ones((3,3))\n",
@@ -761,9 +688,7 @@
{
"cell_type": "markdown",
"id": "19cf9841",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Cost functions\n",
"\n",
@@ -777,10 +702,7 @@
"cell_type": "code",
"execution_count": 6,
"id": "993efa8d",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -815,9 +737,7 @@
{
"cell_type": "markdown",
"id": "011c734c",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Below we give a short example of how these cost function may be used\n",
"to obtain results if you wish to test them out on your own using\n",
@@ -828,10 +748,7 @@
"cell_type": "code",
"execution_count": 7,
"id": "9e1b97f5",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"from autograd import grad\n",
@@ -849,9 +766,7 @@
{
"cell_type": "markdown",
"id": "4acd87b2",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Activation functions\n",
"\n",
@@ -865,10 +780,7 @@
"cell_type": "code",
"execution_count": 8,
"id": "befa86ae",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -923,9 +835,7 @@
{
"cell_type": "markdown",
"id": "c20aa75e",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Below follows a short demonstration of how to use an activation\n",
"function. The derivative of the activation function will be important\n",
@@ -938,10 +848,7 @@
"cell_type": "code",
"execution_count": 9,
"id": "e209f9e5",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"z = np.array([[4, 5, 6]]).T\n",
@@ -959,9 +866,7 @@
{
"cell_type": "markdown",
"id": "524b409b",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### The Neural Network\n",
"\n",
@@ -983,10 +888,7 @@
"cell_type": "code",
"execution_count": 10,
"id": "083116d3",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import math\n",
@@ -1455,9 +1357,7 @@
{
"cell_type": "markdown",
"id": "e4ba55d0",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Before we make a model, we will quickly generate a dataset we can use\n",
"for our linear regression problem as shown below"
@@ -1467,10 +1367,7 @@
"cell_type": "code",
"execution_count": 11,
"id": "c72a22c6",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -1511,9 +1408,7 @@
{
"cell_type": "markdown",
"id": "25b63b47",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Now that we have our dataset ready for the regression, we can create\n",
"our regressor. Note that with the seed parameter, we can make sure our\n",
@@ -1527,10 +1422,7 @@
"cell_type": "code",
"execution_count": 12,
"id": "b6b4e461",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1542,9 +1434,7 @@
{
"cell_type": "markdown",
"id": "d7860b74",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"We then fit our model with our training data using the scheduler of our choice."
]
@@ -1553,10 +1443,7 @@
"cell_type": "code",
"execution_count": 13,
"id": "00522c73",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1568,9 +1455,7 @@
{
"cell_type": "markdown",
"id": "a57bfb12",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Due to the progress bar we can see the MSE (train_error) throughout\n",
"the FFNN's training. Note that the fit() function has some optional\n",
@@ -1584,10 +1469,7 @@
"cell_type": "code",
"execution_count": 14,
"id": "35259e41",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1598,9 +1480,7 @@
{
"cell_type": "markdown",
"id": "4a403370",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"We see that given more epochs to train on, the regressor reaches a lower MSE.\n",
"\n",
@@ -1613,10 +1493,7 @@
"cell_type": "code",
"execution_count": 15,
"id": "6c791130",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_breast_cancer\n",
@@ -1639,10 +1516,7 @@
"cell_type": "code",
"execution_count": 16,
"id": "0ac0258d",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1654,9 +1528,7 @@
{
"cell_type": "markdown",
"id": "a9c74baa",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"We will now make use of our validation data by passing it into our fit function as a keyword argument"
]
@@ -1665,10 +1537,7 @@
"cell_type": "code",
"execution_count": 17,
"id": "e5021bbf",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1680,9 +1549,7 @@
{
"cell_type": "markdown",
"id": "ae0148bb",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Finally, we will create a neural network with 2 hidden layers with activation functions."
]
@@ -1691,10 +1558,7 @@
"cell_type": "code",
"execution_count": 18,
"id": "bb7e4340",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"input_nodes = X_train.shape[1]\n",
@@ -1711,10 +1575,7 @@
"cell_type": "code",
"execution_count": 19,
"id": "d0655afb",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1726,9 +1587,7 @@
{
"cell_type": "markdown",
"id": "dc8a90af",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"### Multiclass classification\n",
"\n",
@@ -1741,10 +1600,7 @@
"cell_type": "code",
"execution_count": 20,
"id": "9305c08a",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_digits\n",
@@ -1778,9 +1634,7 @@
{
"cell_type": "markdown",
"id": "8e208051",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"## Testing the XOR gate and other gates\n",
"\n",
@@ -1791,10 +1645,7 @@
"cell_type": "code",
"execution_count": 21,
"id": "db016d41",
"metadata": {
"collapsed": false,
"editable": true
},
"metadata": {},
"outputs": [],
"source": [
"X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
@@ -1814,15 +1665,31 @@
{
"cell_type": "markdown",
"id": "d1384449",
"metadata": {
"editable": true
},
"metadata": {},
"source": [
"Not bad, but the results depend strongly on the learning reate. Try different learning rates."
]
}
],
"metadata": {},
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/LectureNotes/_build/html/_sources/week41.ipynb
+231 -654
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/_sources/week42.ipynb
+141 -390
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/_sources/week44.ipynb
+1041 -981
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/html/_sources/week45.ipynb
+1671
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/exercisesweek43.html
+16 -6
View File
@@ -343,6 +343,16 @@ const thebe_selector_output = ".output, .cell_output"
Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week44.html">
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week45.html">
Week 45, Recurrent Neural Networks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -933,9 +943,8 @@ Accuracy score on data set: 0.5
Learning rate = 0.0001
Lambda = 0.0001
Accuracy score on data set: 0.5
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
Learning rate = 0.0001
Lambda = 0.001
Accuracy score on data set: 0.5
@@ -1118,7 +1127,7 @@ Accuracy score on data set: 0.5
warnings.warn(
</pre></div>
</div>
<img alt="_images/exercisesweek43_26_3.png" src="_images/exercisesweek43_26_3.png" />
<img alt="_images/exercisesweek43_26_2.png" src="_images/exercisesweek43_26_2.png" />
</div>
</div>
</div>
@@ -5423,8 +5432,9 @@ case.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Adam: Eta=0.001, Lambda=0
[----------------------------------------] 0.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> [----------------------------------------] 0.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> [----------------------------------------] 0.1000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392
doc/LectureNotes/_build/html/genindex.html
+5
View File
@@ -344,6 +344,11 @@ const thebe_selector_output = ".output, .cell_output"
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week45.html">
Week 45, Recurrent Neural Networks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
doc/LectureNotes/_build/html/intro.html
+5
View File
@@ -345,6 +345,11 @@ const thebe_selector_output = ".output, .cell_output"
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week45.html">
Week 45, Recurrent Neural Networks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
doc/LectureNotes/_build/html/objects.inv
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/search.html
+5
View File
@@ -350,6 +350,11 @@ const thebe_selector_output = ".output, .cell_output"
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week45.html">
Week 45, Recurrent Neural Networks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/html/week41.html
+73 -64
View File
@@ -343,6 +343,16 @@ const thebe_selector_output = ".output, .cell_output"
Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week44.html">
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week45.html">
Week 45, Recurrent Neural Networks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -2807,7 +2817,7 @@ the <em>Hadamard product</em>, meaning element-wise multiplication.</p>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Old accuracy on training data: 0.1440501043841336
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3143,7 +3153,7 @@ Lambda = 10.0
Accuracy score on test set: 0.19166666666666668
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3152,7 +3162,7 @@ Lambda = 1e-05
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3161,7 +3171,7 @@ Lambda = 0.0001
Accuracy score on test set: 0.08611111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3170,7 +3180,7 @@ Lambda = 0.001
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3179,7 +3189,7 @@ Lambda = 0.01
Accuracy score on test set: 0.08888888888888889
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3188,7 +3198,7 @@ Lambda = 0.1
Accuracy score on test set: 0.08611111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3197,7 +3207,7 @@ Lambda = 1.0
Accuracy score on test set: 0.08888888888888889
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3206,11 +3216,11 @@ Lambda = 10.0
Accuracy score on test set: 0.09166666666666666
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3219,11 +3229,11 @@ Lambda = 1e-05
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3232,11 +3242,11 @@ Lambda = 0.0001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3245,11 +3255,11 @@ Lambda = 0.001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3258,11 +3268,11 @@ Lambda = 0.01
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3271,7 +3281,7 @@ Lambda = 0.1
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3280,11 +3290,11 @@ Lambda = 1.0
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3293,11 +3303,11 @@ Lambda = 10.0
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3306,11 +3316,11 @@ Lambda = 1e-05
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3319,11 +3329,11 @@ Lambda = 0.0001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3332,11 +3342,11 @@ Lambda = 0.001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3345,11 +3355,11 @@ Lambda = 0.01
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3358,11 +3368,11 @@ Lambda = 0.1
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3371,11 +3381,11 @@ Lambda = 1.0
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -3428,15 +3438,15 @@ Accuracy score on test set: 0.07777777777777778
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19431/953065564.py:4: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74620/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -3753,27 +3763,26 @@ Accuracy score on test set: 0.10555555555555556
Learning rate = 1.0
Lambda = 0.01
Accuracy score on test set: 0.17777777777777778
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 0.1
Accuracy score on test set: 0.08333333333333333
Learning rate = 1.0
Lambda = 1.0
Accuracy score on test set: 0.08888888888888889
Lambda = 0.1
Accuracy score on test set: 0.08333333333333333
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 1.0
Accuracy score on test set: 0.08888888888888889
Learning rate = 1.0
Lambda = 10.0
Accuracy score on test set: 0.09444444444444444
Learning rate = 10.0
Lambda = 1e-05
Accuracy score on test set: 0.17222222222222222
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
Lambda = 1e-05
Accuracy score on test set: 0.17222222222222222
Learning rate = 10.0
Lambda = 0.0001
Accuracy score on test set: 0.11666666666666667
doc/LectureNotes/_build/html/week42.html
+95 -88
View File
@@ -343,6 +343,16 @@ const thebe_selector_output = ".output, .cell_output"
Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week44.html">
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week45.html">
Week 45, Recurrent Neural Networks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -1687,7 +1697,7 @@ the <em>Hadamard product</em>, meaning element-wise multiplication.</p>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Old accuracy on training data: 0.1440501043841336
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2023,7 +2033,7 @@ Lambda = 10.0
Accuracy score on test set: 0.19166666666666668
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2032,7 +2042,7 @@ Lambda = 1e-05
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2041,7 +2051,7 @@ Lambda = 0.0001
Accuracy score on test set: 0.08611111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2050,7 +2060,7 @@ Lambda = 0.001
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2059,7 +2069,7 @@ Lambda = 0.01
Accuracy score on test set: 0.08888888888888889
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2068,7 +2078,7 @@ Lambda = 0.1
Accuracy score on test set: 0.08611111111111111
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2077,7 +2087,7 @@ Lambda = 1.0
Accuracy score on test set: 0.08888888888888889
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2086,11 +2096,11 @@ Lambda = 10.0
Accuracy score on test set: 0.09166666666666666
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2099,11 +2109,11 @@ Lambda = 1e-05
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2112,11 +2122,11 @@ Lambda = 0.0001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2125,11 +2135,11 @@ Lambda = 0.001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2138,11 +2148,11 @@ Lambda = 0.01
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2151,7 +2161,7 @@ Lambda = 0.1
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2160,11 +2170,11 @@ Lambda = 1.0
Accuracy score on test set: 0.10555555555555556
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2173,11 +2183,11 @@ Lambda = 10.0
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2186,11 +2196,11 @@ Lambda = 1e-05
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2199,11 +2209,11 @@ Lambda = 0.0001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2212,11 +2222,11 @@ Lambda = 0.001
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2225,11 +2235,11 @@ Lambda = 0.01
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2238,11 +2248,11 @@ Lambda = 0.1
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2251,11 +2261,11 @@ Lambda = 1.0
Accuracy score on test set: 0.07777777777777778
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:43: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:43: RuntimeWarning: overflow encountered in exp
exp_term = np.exp(self.z_o)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
</pre></div>
</div>
@@ -2308,15 +2318,15 @@ Accuracy score on test set: 0.07777777777777778
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19440/953065564.py:4: RuntimeWarning: overflow encountered in exp
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_74630/953065564.py:4: RuntimeWarning: overflow encountered in exp
return 1/(1 + np.exp(-x))
</pre></div>
</div>
@@ -2607,17 +2617,18 @@ Accuracy score on test set: 0.9055555555555556
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 0.1
Accuracy score on test set: 0.8805555555555555
Learning rate = 0.1
Lambda = 1.0
Accuracy score on test set: 0.8722222222222222
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
Lambda = 1.0
Accuracy score on test set: 0.8722222222222222
Learning rate = 0.1
Lambda = 10.0
Accuracy score on test set: 0.8666666666666667
Learning rate = 1.0
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
Lambda = 1e-05
Accuracy score on test set: 0.08611111111111111
@@ -3097,34 +3108,8 @@ Accuracy score on data set: 0.5
Learning rate = 1.0
Lambda = 1.0
Accuracy score on data set: 0.5
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate =
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1.0
Learning rate = 1.0
Lambda = 10.0
Accuracy score on data set: 0.5
@@ -3157,7 +3142,29 @@ Lambda = 10.0
Accuracy score on data set: 0.5
</pre></div>
</div>
<img alt="_images/week42_88_4.png" src="_images/week42_88_4.png" />
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<img alt="_images/week42_88_2.png" src="_images/week42_88_2.png" />
</div>
</div>
</div>
doc/LectureNotes/_build/html/week44.html
+229 -26
View File
@@ -55,7 +55,7 @@ const thebe_selector_output = ".output, .cell_output"
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Project 1 on Machine Learning, deadline October 9 (midnight), 2023" href="project1.html" />
<link rel="next" title="Week 45, Recurrent Neural Networks" href="week45.html" />
<link rel="prev" title="Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations" href="week43.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="None">
@@ -348,6 +348,11 @@ const thebe_selector_output = ".output, .cell_output"
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week45.html">
Week 45, Recurrent Neural Networks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -1206,7 +1211,9 @@ doconce format html week44.do.txt --no_mako -->
<li><p>Exercise on writing your own neural network code, application to the OR and XOR gates, see notes from last week</p></li>
<li><p>The exercise this week is a continuation from last week</p></li>
<li><p>Discussion of project 2</p></li>
<li><p><a class="reference external" href="https://youtu.be/Ia6wwDLxqtM">Video of lab session from last week</a></p></li>
<li><p><a class="reference external" href="https://youtu.be/Ia6wwDLxqtM">Video of lab session from week 43</a></p></li>
<li><p><a class="reference external" href="https://youtu.be/EajWMW__k0I">Video of lab session from week 44</a></p></li>
<li><p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/Exercisesweek44.pdf">See also whiteboard notes from lab session week 44</a></p></li>
</ul>
<p><strong>Material for the lecture on Thursday November 2, 2023.</strong></p>
<ul class="simple">
@@ -2228,7 +2235,7 @@ labels = (n_inputs) = (1797,)
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/optimizer_v2/gradient_descent.py:102: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
super(SGD, self).__init__(name, **kwargs)
2023-10-30 10:31:09.117456: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2023-11-06 06:34:51.825607: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
</pre></div>
</div>
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
@@ -2373,6 +2380,112 @@ labels = (n_inputs) = (1797,)
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>45/45 [==============================] - 4s 83ms/step - loss: 3.3532 - accuracy: 0.1134
12/12 [==============================] - 1s 66ms/step - loss: 3.4256 - accuracy: 0.0917
45/45 [==============================] - 4s 78ms/step - loss: 3.3619 - accuracy: 0.1141
12/12 [==============================] - 1s 64ms/step - loss: 3.4338 - accuracy: 0.0944
45/45 [==============================] - 3s 76ms/step - loss: 3.4466 - accuracy: 0.1141
12/12 [==============================] - 1s 62ms/step - loss: 3.5186 - accuracy: 0.0944
45/45 [==============================] - 4s 78ms/step - loss: 4.2927 - accuracy: 0.1141
12/12 [==============================] - 1s 66ms/step - loss: 4.3646 - accuracy: 0.0944
45/45 [==============================] - 3s 76ms/step - loss: 12.7049 - accuracy: 0.1141
12/12 [==============================] - 1s 63ms/step - loss: 12.7761 - accuracy: 0.0944
45/45 [==============================] - 3s 74ms/step - loss: 91.9603 - accuracy: 0.1134
12/12 [==============================] - 1s 59ms/step - loss: 92.0269 - accuracy: 0.0944
45/45 [==============================] - 3s 76ms/step - loss: 519.9026 - accuracy: 0.1148
12/12 [==============================] - 1s 62ms/step - loss: 519.9384 - accuracy: 0.0972
45/45 [==============================] - 4s 79ms/step - loss: 1.4512 - accuracy: 0.5449
12/12 [==============================] - 1s 62ms/step - loss: 1.5349 - accuracy: 0.4694
45/45 [==============================] - 3s 75ms/step - loss: 1.4605 - accuracy: 0.5470
12/12 [==============================] - 1s 62ms/step - loss: 1.5442 - accuracy: 0.4667
45/45 [==============================] - 4s 80ms/step - loss: 1.5454 - accuracy: 0.5477
12/12 [==============================] - 1s 64ms/step - loss: 1.6288 - accuracy: 0.4667
45/45 [==============================] - 3s 76ms/step - loss: 2.3881 - accuracy: 0.5435
12/12 [==============================] - 1s 68ms/step - loss: 2.4714 - accuracy: 0.4722
45/45 [==============================] - 4s 80ms/step - loss: 10.3364 - accuracy: 0.5393
12/12 [==============================] - 1s 63ms/step - loss: 10.4141 - accuracy: 0.4556
45/45 [==============================] - 4s 79ms/step - loss: 53.4810 - accuracy: 0.4983
12/12 [==============================] - 1s 65ms/step - loss: 53.5240 - accuracy: 0.4472
45/45 [==============================] - 3s 76ms/step - loss: 4.6258 - accuracy: 0.1044
12/12 [==============================] - 1s 66ms/step - loss: 4.6259 - accuracy: 0.0889
45/45 [==============================] - 4s 80ms/step - loss: 0.1946 - accuracy: 0.9534
12/12 [==============================] - 1s 67ms/step - loss: 0.2641 - accuracy: 0.9194
45/45 [==============================] - 3s 77ms/step - loss: 0.2027 - accuracy: 0.9541
12/12 [==============================] - 1s 64ms/step - loss: 0.2736 - accuracy: 0.9167
45/45 [==============================] - 4s 79ms/step - loss: 0.2901 - accuracy: 0.9541
12/12 [==============================] - 1s 65ms/step - loss: 0.3604 - accuracy: 0.9167
45/45 [==============================] - 3s 77ms/step - loss: 1.1123 - accuracy: 0.9520
12/12 [==============================] - 1s 58ms/step - loss: 1.1848 - accuracy: 0.9167
45/45 [==============================] - 4s 81ms/step - loss: 5.7392 - accuracy: 0.9415
12/12 [==============================] - 1s 61ms/step - loss: 5.7980 - accuracy: 0.9250
45/45 [==============================] - 3s 77ms/step - loss: 2.5993 - accuracy: 0.4085
12/12 [==============================] - 1s 65ms/step - loss: 2.6003 - accuracy: 0.3472
45/45 [==============================] - 3s 77ms/step - loss: 2.3024 - accuracy: 0.1044
12/12 [==============================] - 1s 66ms/step - loss: 2.3032 - accuracy: 0.0889
45/45 [==============================] - 4s 79ms/step - loss: 0.0180 - accuracy: 1.0000
12/12 [==============================] - 1s 66ms/step - loss: 0.0958 - accuracy: 0.9694
45/45 [==============================] - 4s 80ms/step - loss: 0.0280 - accuracy: 0.9986
12/12 [==============================] - 1s 68ms/step - loss: 0.1060 - accuracy: 0.9750
45/45 [==============================] - 3s 77ms/step - loss: 0.1148 - accuracy: 0.9986
12/12 [==============================] - 1s 67ms/step - loss: 0.1862 - accuracy: 0.9778
45/45 [==============================] - 3s 76ms/step - loss: 0.6357 - accuracy: 0.9958
12/12 [==============================] - 1s 67ms/step - loss: 0.6819 - accuracy: 0.9750
45/45 [==============================] - 3s 76ms/step - loss: 0.9286 - accuracy: 0.9499
12/12 [==============================] - 1s 65ms/step - loss: 0.9978 - accuracy: 0.9028
45/45 [==============================] - 3s 75ms/step - loss: 2.3020 - accuracy: 0.1044
12/12 [==============================] - 1s 64ms/step - loss: 2.3064 - accuracy: 0.0889
45/45 [==============================] - 4s 78ms/step - loss: 2.3020 - accuracy: 0.1044
12/12 [==============================] - 1s 66ms/step - loss: 2.3065 - accuracy: 0.0889
45/45 [==============================] - 4s 78ms/step - loss: 0.0060 - accuracy: 1.0000
12/12 [==============================] - 1s 62ms/step - loss: 0.2141 - accuracy: 0.9528
45/45 [==============================] - 3s 76ms/step - loss: 0.0368 - accuracy: 0.9930
12/12 [==============================] - 1s 65ms/step - loss: 0.2714 - accuracy: 0.9472
45/45 [==============================] - 4s 80ms/step - loss: 0.1343 - accuracy: 0.9910
12/12 [==============================] - 1s 68ms/step - loss: 0.2996 - accuracy: 0.9556
45/45 [==============================] - 4s 78ms/step - loss: 0.4220 - accuracy: 0.9207
12/12 [==============================] - 1s 66ms/step - loss: 0.6088 - accuracy: 0.8611
45/45 [==============================] - 4s 79ms/step - loss: 1.6795 - accuracy: 0.6764
12/12 [==============================] - 1s 61ms/step - loss: 1.7069 - accuracy: 0.6556
45/45 [==============================] - 4s 79ms/step - loss: 2.3020 - accuracy: 0.1044
12/12 [==============================] - 1s 67ms/step - loss: 2.3077 - accuracy: 0.0778
45/45 [==============================] - 4s 80ms/step - loss: nan - accuracy: 0.1044
12/12 [==============================] - 1s 68ms/step - loss: nan - accuracy: 0.0778
45/45 [==============================] - 4s 81ms/step - loss: 21.9981 - accuracy: 0.1044
12/12 [==============================] - 1s 67ms/step - loss: 22.0022 - accuracy: 0.0778
45/45 [==============================] - 4s 79ms/step - loss: 44.1977 - accuracy: 0.1044
12/12 [==============================] - 1s 67ms/step - loss: 44.2070 - accuracy: 0.0778
45/45 [==============================] - 3s 77ms/step - loss: 6.3493 - accuracy: 0.1044
12/12 [==============================] - 1s 68ms/step - loss: 6.3536 - accuracy: 0.0778
45/45 [==============================] - 4s 80ms/step - loss: 2.3030 - accuracy: 0.1044
12/12 [==============================] - 1s 66ms/step - loss: 2.3082 - accuracy: 0.0889
45/45 [==============================] - 4s 79ms/step - loss: 2.3029 - accuracy: 0.1044
12/12 [==============================] - 1s 66ms/step - loss: 2.3126 - accuracy: 0.0778
45/45 [==============================] - 4s 78ms/step - loss: nan - accuracy: 0.1044
12/12 [==============================] - 1s 64ms/step - loss: nan - accuracy: 0.0778
45/45 [==============================] - 4s 79ms/step - loss: nan - accuracy: 0.1044
12/12 [==============================] - 1s 66ms/step - loss: nan - accuracy: 0.0778
45/45 [==============================] - 4s 79ms/step - loss: 6130353.0000 - accuracy: 0.1009
12/12 [==============================] - 1s 66ms/step - loss: 6130353.0000 - accuracy: 0.1056
45/45 [==============================] - 3s 77ms/step - loss: 388451.4375 - accuracy: 0.1016
12/12 [==============================] - 1s 68ms/step - loss: 388451.5000 - accuracy: 0.0917
45/45 [==============================] - 4s 80ms/step - loss: 2.4241 - accuracy: 0.1044
12/12 [==============================] - 1s 66ms/step - loss: 2.4314 - accuracy: 0.0889
45/45 [==============================] - 4s 81ms/step - loss: 2.4394 - accuracy: 0.1037
12/12 [==============================] - 1s 62ms/step - loss: 2.5014 - accuracy: 0.0889
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>45/45 [==============================] - 4s 79ms/step - loss: nan - accuracy: 0.1044
12/12 [==============================] - 1s 65ms/step - loss: nan - accuracy: 0.0778
45/45 [==============================] - 4s 81ms/step - loss: nan - accuracy: 0.1044
12/12 [==============================] - 1s 64ms/step - loss: nan - accuracy: 0.0778
45/45 [==============================] - 4s 78ms/step - loss: nan - accuracy: 0.1044
12/12 [==============================] - 1s 67ms/step - loss: nan - accuracy: 0.0778
</pre></div>
</div>
<img alt="_images/week44_135_2.png" src="_images/week44_135_2.png" />
<img alt="_images/week44_135_3.png" src="_images/week44_135_3.png" />
</div>
</div>
</div>
<div class="section" id="the-cifar01-data-set">
@@ -2403,23 +2516,26 @@ exclusive and there is no overlap between them.</p>
<p>To verify that the dataset looks correct, let’s plot the first 25 images from the training set and display the class name below each image.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span>class_names = [&#39;airplane&#39;, &#39;automobile&#39;, &#39;bird&#39;, &#39;cat&#39;, &#39;deer&#39;,
&#39;dog&#39;, &#39;frog&#39;, &#39;horse&#39;, &#39;ship&#39;, &#39;truck&#39;]
​
plt.figure(figsize=(10,10))
for i in range(25):
plt.subplot(5,5,i+1)
plt.xticks([])
plt.yticks([])
plt.grid(False)
plt.imshow(train_images[i], cmap=plt.cm.binary)
# The CIFAR labels happen to be arrays,
# which is why you need the extra index
plt.xlabel(class_names[train_labels[i][0]])
plt.show()
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">class_names</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;airplane&#39;</span><span class="p">,</span> <span class="s1">&#39;automobile&#39;</span><span class="p">,</span> <span class="s1">&#39;bird&#39;</span><span class="p">,</span> <span class="s1">&#39;cat&#39;</span><span class="p">,</span> <span class="s1">&#39;deer&#39;</span><span class="p">,</span>
<span class="s1">&#39;dog&#39;</span><span class="p">,</span> <span class="s1">&#39;frog&#39;</span><span class="p">,</span> <span class="s1">&#39;horse&#39;</span><span class="p">,</span> <span class="s1">&#39;ship&#39;</span><span class="p">,</span> <span class="s1">&#39;truck&#39;</span><span class="p">]</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">25</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xticks</span><span class="p">([])</span>
<span class="n">plt</span><span class="o">.</span><span class="n">yticks</span><span class="p">([])</span>
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="kc">False</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">train_images</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">cmap</span><span class="o">=</span><span class="n">plt</span><span class="o">.</span><span class="n">cm</span><span class="o">.</span><span class="n">binary</span><span class="p">)</span>
<span class="c1"># The CIFAR labels happen to be arrays, </span>
<span class="c1"># which is why you need the extra index</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="n">class_names</span><span class="p">[</span><span class="n">train_labels</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]])</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/week44_139_0.png" src="_images/week44_139_0.png" />
</div>
</div>
</div>
<div class="section" id="set-up-the-model">
@@ -2441,6 +2557,31 @@ plt.show()
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Model: &quot;sequential_49&quot;
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_49 (Conv2D) (None, 30, 30, 32) 896
max_pooling2d_49 (MaxPoolin (None, 15, 15, 32) 0
g2D)
conv2d_50 (Conv2D) (None, 13, 13, 64) 18496
max_pooling2d_50 (MaxPoolin (None, 6, 6, 64) 0
g2D)
conv2d_51 (Conv2D) (None, 4, 4, 64) 36928
=================================================================
Total params: 56,320
Trainable params: 56,320
Non-trainable params: 0
_________________________________________________________________
</pre></div>
</div>
</div>
</div>
<p>You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer.</p>
</div>
@@ -2458,12 +2599,43 @@ layer with 10 outputs and a softmax activation.</p>
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Flatten</span><span class="p">())</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Dense</span><span class="p">(</span><span class="mi">64</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;relu&#39;</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Dense</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
<span class="n">Here</span><span class="s1">&#39;s the complete architecture of our model.</span>
<span class="c1">#Here&#39;s the complete architecture of our model.</span>
<span class="n">model</span><span class="o">.</span><span class="n">summary</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Model: &quot;sequential_49&quot;
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_49 (Conv2D) (None, 30, 30, 32) 896
max_pooling2d_49 (MaxPoolin (None, 15, 15, 32) 0
g2D)
conv2d_50 (Conv2D) (None, 13, 13, 64) 18496
max_pooling2d_50 (MaxPoolin (None, 6, 6, 64) 0
g2D)
conv2d_51 (Conv2D) (None, 4, 4, 64) 36928
flatten_49 (Flatten) (None, 1024) 0
dense_98 (Dense) (None, 64) 65600
dense_99 (Dense) (None, 10) 650
=================================================================
Total params: 122,570
Trainable params: 122,570
Non-trainable params: 0
_________________________________________________________________
</pre></div>
</div>
</div>
</div>
<p>As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers.</p>
</div>
@@ -2471,12 +2643,36 @@ layer with 10 outputs and a softmax activation.</p>
<h2>Compile and train the model<a class="headerlink" href="#compile-and-train-the-model" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span>model.compile(optimizer=&#39;adam&#39;,
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[&#39;accuracy&#39;])
​
history = model.fit(train_images, train_labels, epochs=10,
validation_data=(test_images, test_labels))
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">optimizer</span><span class="o">=</span><span class="s1">&#39;adam&#39;</span><span class="p">,</span>
<span class="n">loss</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">keras</span><span class="o">.</span><span class="n">losses</span><span class="o">.</span><span class="n">SparseCategoricalCrossentropy</span><span class="p">(</span><span class="n">from_logits</span><span class="o">=</span><span class="kc">True</span><span class="p">),</span>
<span class="n">metrics</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;accuracy&#39;</span><span class="p">])</span>
<span class="n">history</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">train_images</span><span class="p">,</span> <span class="n">train_labels</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span>
<span class="n">validation_data</span><span class="o">=</span><span class="p">(</span><span class="n">test_images</span><span class="p">,</span> <span class="n">test_labels</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 1/10
1563/1563 [==============================] - 28s 15ms/step - loss: 1.5353 - accuracy: 0.4394 - val_loss: 1.2184 - val_accuracy: 0.5616
Epoch 2/10
1563/1563 [==============================] - 18s 12ms/step - loss: 1.1560 - accuracy: 0.5909 - val_loss: 1.0712 - val_accuracy: 0.6214
Epoch 3/10
1563/1563 [==============================] - 20s 13ms/step - loss: 1.0176 - accuracy: 0.6411 - val_loss: 1.0012 - val_accuracy: 0.6527
Epoch 4/10
1563/1563 [==============================] - 18s 12ms/step - loss: 0.9234 - accuracy: 0.6788 - val_loss: 0.9674 - val_accuracy: 0.6599
Epoch 5/10
1563/1563 [==============================] - 18s 12ms/step - loss: 0.8524 - accuracy: 0.7033 - val_loss: 0.8982 - val_accuracy: 0.6890
Epoch 6/10
1563/1563 [==============================] - 18s 11ms/step - loss: 0.7966 - accuracy: 0.7203 - val_loss: 0.9145 - val_accuracy: 0.6835
Epoch 7/10
1563/1563 [==============================] - 18s 11ms/step - loss: 0.7483 - accuracy: 0.7407 - val_loss: 0.9275 - val_accuracy: 0.6849
Epoch 8/10
1563/1563 [==============================] - 17s 11ms/step - loss: 0.7049 - accuracy: 0.7532 - val_loss: 0.9460 - val_accuracy: 0.6781
Epoch 9/10
1563/1563 [==============================] - 21s 13ms/step - loss: 0.6663 - accuracy: 0.7696 - val_loss: 0.8528 - val_accuracy: 0.7078
Epoch 10/10
1563/1563 [==============================] - 19s 12ms/step - loss: 0.6297 - accuracy: 0.7788 - val_loss: 0.8747 - val_accuracy: 0.7032
</pre></div>
</div>
</div>
@@ -2499,6 +2695,13 @@ history = model.fit(train_images, train_labels, epochs=10,
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>313/313 - 2s - loss: 0.8747 - accuracy: 0.7032 - 2s/epoch - 5ms/step
0.7031999826431274
</pre></div>
</div>
<img alt="_images/week44_149_1.png" src="_images/week44_149_1.png" />
</div>
</div>
</div>
<div class="section" id="building-our-own-cnn-code">
@@ -5318,10 +5521,10 @@ optimization technique.</p>
<p class="prev-next-title">Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations</p>
</div>
</a>
<a class='right-next' id="next-link" href="project1.html" title="next page">
<a class='right-next' id="next-link" href="week45.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Project 1 on Machine Learning, deadline October 9 (midnight), 2023</p>
<p class="prev-next-title">Week 45, Recurrent Neural Networks</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
doc/LectureNotes/_build/html/week45.html
+2690
View File
File diff suppressed because it is too large Load Diff