update of p2
This commit is contained in:
Morten Hjorth-Jensen
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "061af572",
|
||||
"id": "96e577ca",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "22578683",
|
||||
"id": "067c02b9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "61fb162f",
|
||||
"id": "01f9fedd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -58,7 +58,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "104c69e1",
|
||||
"id": "9f8e4871",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -104,7 +104,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0d2c42e3",
|
||||
"id": "460cc6ea",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -121,7 +121,7 @@
|
||||
"\n",
|
||||
"* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be\n",
|
||||
"\n",
|
||||
" * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
" * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
"\n",
|
||||
"* Classification.\n",
|
||||
"\n",
|
||||
@@ -132,7 +132,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8baab67",
|
||||
"id": "d62a07ef",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -162,7 +162,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "87e7ed71",
|
||||
"id": "9cd8b8ac",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -189,7 +189,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5a26b6ad",
|
||||
"id": "5931b155",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -205,7 +205,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "096fe6c4",
|
||||
"id": "b273fc8a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -217,7 +217,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd986596",
|
||||
"id": "e13db1ec",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -252,7 +252,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e853d4b6",
|
||||
"id": "4f864e31",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -270,7 +270,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fc2d413b",
|
||||
"id": "c9faeafd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -285,7 +285,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e6821051",
|
||||
"id": "d865c22b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -302,7 +302,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cba72d68",
|
||||
"id": "5270af8f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -328,7 +328,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "e16fb528",
|
||||
"id": "4e0e1fea",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -347,7 +347,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "73599f42",
|
||||
"id": "8fe85677",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -358,7 +358,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "f1a639ef",
|
||||
"id": "b28318b2",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -370,7 +370,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90fb7b41",
|
||||
"id": "97e02c71",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -381,7 +381,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "424af629",
|
||||
"id": "88af355c",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -394,7 +394,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3c006080",
|
||||
"id": "d1f8f0ed",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -407,7 +407,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a18ddd54",
|
||||
"id": "554b3a48",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -419,7 +419,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1a1afaf9",
|
||||
"id": "77bfdd5c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -434,7 +434,7 @@
|
||||
"code for classification and pertinent results against a similar code using **Scikit-Learn** or **tensorflow/keras** or **pytorch**.\n",
|
||||
"\n",
|
||||
"If you have time, you can use the functionality of **scikit-learn** and compare your neural network results with those from Logistic regression. This is optional.\n",
|
||||
"The weblink here <https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3>compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero. \n",
|
||||
"The weblink here <https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3>compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero and keeping just the input and the output layers. \n",
|
||||
"\n",
|
||||
"If you wish to compare with say Logisti Regression from **scikit-learn**, the following code uses the above data set"
|
||||
]
|
||||
@@ -442,7 +442,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "3c37cbaf",
|
||||
"id": "eaa9e72e",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -464,7 +464,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "106b9303",
|
||||
"id": "c7ba883e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -480,7 +480,88 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "55da0d7f",
|
||||
"id": "595be693",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Summary of methods to implement and analyze\n",
|
||||
"\n",
|
||||
"**Required Implementation:**\n",
|
||||
"1. Reuse the regression code and results from project 1, these will act as a benchmark for seeing how suited a neural network is for this regression task.\n",
|
||||
"\n",
|
||||
"2. Implement a neural network with\n",
|
||||
"\n",
|
||||
" * A flexible number of layers\n",
|
||||
"\n",
|
||||
" * A flexible number of nodes in each layer\n",
|
||||
"\n",
|
||||
" * A changeable activation function in each layer (Sigmoid, ReLU, LeakyReLU, as well as Linear and Softmax)\n",
|
||||
"\n",
|
||||
" * A changeable cost function, which will be set to MSE for regression and cross-entropy for multiple-classification\n",
|
||||
"\n",
|
||||
" * An optional L1 or L2 norm of the weights and biases in the cost function (only used for computing gradients, not interpretable metrics)\n",
|
||||
"\n",
|
||||
"3. Implement the back-propagation algorithm to compute the gradient of your neural network\n",
|
||||
"\n",
|
||||
"4. Reuse the implementation of Plain and Stochastic Gradient Descent from Project 1 (and adapt the code to work with the your neural network)\n",
|
||||
"\n",
|
||||
" * With no optimization algorithm\n",
|
||||
"\n",
|
||||
" * With RMS Prop\n",
|
||||
"\n",
|
||||
" * With ADAM\n",
|
||||
"\n",
|
||||
"5. Implement scaling and train-test splitting of your data, preferably using sklearn\n",
|
||||
"\n",
|
||||
"6. Implement and compute metrics like the MSE and Accuracy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "35138b41",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Required Analysis:\n",
|
||||
"\n",
|
||||
"1. Briefly show and argue for the advantages and disadvantages of the methods from Project 1.\n",
|
||||
"\n",
|
||||
"2. Explore and show the impact of changing the number of layers, nodes per layer, choice of activation function, and inclusion of L1 and L2 norms. Present only the most interesting results from this exploration. 2D Heatmaps will be good for this: Start with finding a well performing set of hyper-parameters, then change two at a time in a range that shows good and bad performance.\n",
|
||||
"\n",
|
||||
"3. Show and argue for the advantages and disadvantages of using a neural network for regression on your data\n",
|
||||
"\n",
|
||||
"4. Show and argue for the advantages and disadvantages of using a neural network for classification on your data\n",
|
||||
"\n",
|
||||
"5. Show and argue for the advantages and disadvantages of the different gradient methods and learning rates when training the neural network"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b18bea03",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Optional (Note that you should include at least two of these in the report):\n",
|
||||
"\n",
|
||||
"1. Implement Logistic Regression as simple classification model case (equivalent to a Neural Network with just the output layer)\n",
|
||||
"\n",
|
||||
"2. Compute the gradient of the neural network with autograd, to show that it gives the same result as your hand-written backpropagation.\n",
|
||||
"\n",
|
||||
"3. Compare your results with results from using a machine-learning library like pytorch (https://docs.pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html)\n",
|
||||
"\n",
|
||||
"4. Use a more complex classification dataset instead, like the fashion MNIST (see <https://www.kaggle.com/datasets/zalando-research/fashionmnist>)\n",
|
||||
"\n",
|
||||
"5. Use a more complex regression dataset instead, like the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
"\n",
|
||||
"6. Compute and interpret a confusion matrix of your best classification model (see <https://www.researchgate.net/figure/Confusion-matrix-of-MNIST-and-F-MNIST-embeddings_fig5_349758607>)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "580d8424",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -496,7 +577,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d3731e2c",
|
||||
"id": "96f5c67e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -527,7 +608,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6c7c5340",
|
||||
"id": "d1bc28ba",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -424,6 +424,11 @@ document.write(`
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#part-g-critical-evaluation-of-the-various-algorithms">Part g) Critical evaluation of the various algorithms</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#summary-of-methods-to-implement-and-analyze">Summary of methods to implement and analyze</a><ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#required-analysis">Required Analysis:</a></li>
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#optional-note-that-you-should-include-at-least-two-of-these-in-the-report">Optional (Note that you should include at least two of these in the report):</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#background-literature">Background literature</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#introduction-to-numerical-projects">Introduction to numerical projects</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#format-for-electronic-delivery-of-report-and-programs">Format for electronic delivery of report and programs</a></li>
|
||||
@@ -510,7 +515,7 @@ feed-forward neural network (FFNN) code. The exercises from week 41 and 42 (see
|
||||
<ul class="simple">
|
||||
<li><p>Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be</p>
|
||||
<ul>
|
||||
<li><p>The simple one-dimensional function Runge function from project 1, that is <span class="math notranslate nohighlight">\(f(x) = \frac{1}{1+25x^2}\)</span>. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the the two-dimensional Runge function <span class="math notranslate nohighlight">\(f(x,y)=\left[(10x - 5)^2 + (10y - 5)^2 + 1 \right]^{-1}\)</span>, or even more complicated two-dimensional functions (see the supplementary material of <a class="reference external" href="https://www.nature.com/articles/s41467-025-61362-4">https://www.nature.com/articles/s41467-025-61362-4</a> for an extensive list of two-dimensional functions).</p></li>
|
||||
<li><p>The simple one-dimensional function Runge function from project 1, that is <span class="math notranslate nohighlight">\(f(x) = \frac{1}{1+25x^2}\)</span>. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the two-dimensional Runge function <span class="math notranslate nohighlight">\(f(x,y)=\left[(10x - 5)^2 + (10y - 5)^2 + 1 \right]^{-1}\)</span>, or even more complicated two-dimensional functions (see the supplementary material of <a class="reference external" href="https://www.nature.com/articles/s41467-025-61362-4">https://www.nature.com/articles/s41467-025-61362-4</a> for an extensive list of two-dimensional functions).</p></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><p>Classification.</p></li>
|
||||
@@ -670,7 +675,7 @@ represents the target and <span class="math notranslate nohighlight">\(y_i\)</sp
|
||||
<p>Again, we strongly recommend that you compare your own neural Network
|
||||
code for classification and pertinent results against a similar code using <strong>Scikit-Learn</strong> or <strong>tensorflow/keras</strong> or <strong>pytorch</strong>.</p>
|
||||
<p>If you have time, you can use the functionality of <strong>scikit-learn</strong> and compare your neural network results with those from Logistic regression. This is optional.
|
||||
The weblink here <a class="reference external" href="https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3">https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3</a>compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero.</p>
|
||||
The weblink here <a class="reference external" href="https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3">https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3</a>compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero and keeping just the input and the output layers.</p>
|
||||
<p>If you wish to compare with say Logisti Regression from <strong>scikit-learn</strong>, the following code uses the above data set</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -699,6 +704,55 @@ is best for the classification case. These codes can also be part of
|
||||
your final project 3, but now applied to other data sets.</p>
|
||||
</section>
|
||||
</section>
|
||||
<section id="summary-of-methods-to-implement-and-analyze">
|
||||
<h2>Summary of methods to implement and analyze<a class="headerlink" href="#summary-of-methods-to-implement-and-analyze" title="Link to this heading">#</a></h2>
|
||||
<p><strong>Required Implementation:</strong></p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>Reuse the regression code and results from project 1, these will act as a benchmark for seeing how suited a neural network is for this regression task.</p></li>
|
||||
<li><p>Implement a neural network with</p></li>
|
||||
</ol>
|
||||
<ul class="simple">
|
||||
<li><p>A flexible number of layers</p></li>
|
||||
<li><p>A flexible number of nodes in each layer</p></li>
|
||||
<li><p>A changeable activation function in each layer (Sigmoid, ReLU, LeakyReLU, as well as Linear and Softmax)</p></li>
|
||||
<li><p>A changeable cost function, which will be set to MSE for regression and cross-entropy for multiple-classification</p></li>
|
||||
<li><p>An optional L1 or L2 norm of the weights and biases in the cost function (only used for computing gradients, not interpretable metrics)</p></li>
|
||||
</ul>
|
||||
<ol class="arabic simple" start="3">
|
||||
<li><p>Implement the back-propagation algorithm to compute the gradient of your neural network</p></li>
|
||||
<li><p>Reuse the implementation of Plain and Stochastic Gradient Descent from Project 1 (and adapt the code to work with the your neural network)</p></li>
|
||||
</ol>
|
||||
<ul class="simple">
|
||||
<li><p>With no optimization algorithm</p></li>
|
||||
<li><p>With RMS Prop</p></li>
|
||||
<li><p>With ADAM</p></li>
|
||||
</ul>
|
||||
<ol class="arabic simple" start="5">
|
||||
<li><p>Implement scaling and train-test splitting of your data, preferably using sklearn</p></li>
|
||||
<li><p>Implement and compute metrics like the MSE and Accuracy</p></li>
|
||||
</ol>
|
||||
<section id="required-analysis">
|
||||
<h3>Required Analysis:<a class="headerlink" href="#required-analysis" title="Link to this heading">#</a></h3>
|
||||
<ol class="arabic simple">
|
||||
<li><p>Briefly show and argue for the advantages and disadvantages of the methods from Project 1.</p></li>
|
||||
<li><p>Explore and show the impact of changing the number of layers, nodes per layer, choice of activation function, and inclusion of L1 and L2 norms. Present only the most interesting results from this exploration. 2D Heatmaps will be good for this: Start with finding a well performing set of hyper-parameters, then change two at a time in a range that shows good and bad performance.</p></li>
|
||||
<li><p>Show and argue for the advantages and disadvantages of using a neural network for regression on your data</p></li>
|
||||
<li><p>Show and argue for the advantages and disadvantages of using a neural network for classification on your data</p></li>
|
||||
<li><p>Show and argue for the advantages and disadvantages of the different gradient methods and learning rates when training the neural network</p></li>
|
||||
</ol>
|
||||
</section>
|
||||
<section id="optional-note-that-you-should-include-at-least-two-of-these-in-the-report">
|
||||
<h3>Optional (Note that you should include at least two of these in the report):<a class="headerlink" href="#optional-note-that-you-should-include-at-least-two-of-these-in-the-report" title="Link to this heading">#</a></h3>
|
||||
<ol class="arabic simple">
|
||||
<li><p>Implement Logistic Regression as simple classification model case (equivalent to a Neural Network with just the output layer)</p></li>
|
||||
<li><p>Compute the gradient of the neural network with autograd, to show that it gives the same result as your hand-written backpropagation.</p></li>
|
||||
<li><p>Compare your results with results from using a machine-learning library like pytorch (<a class="reference external" href="https://docs.pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html">https://docs.pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html</a>)</p></li>
|
||||
<li><p>Use a more complex classification dataset instead, like the fashion MNIST (see <a class="reference external" href="https://www.kaggle.com/datasets/zalando-research/fashionmnist">https://www.kaggle.com/datasets/zalando-research/fashionmnist</a>)</p></li>
|
||||
<li><p>Use a more complex regression dataset instead, like the two-dimensional Runge function <span class="math notranslate nohighlight">\(f(x,y)=\left[(10x - 5)^2 + (10y - 5)^2 + 1 \right]^{-1}\)</span>, or even more complicated two-dimensional functions (see the supplementary material of <a class="reference external" href="https://www.nature.com/articles/s41467-025-61362-4">https://www.nature.com/articles/s41467-025-61362-4</a> for an extensive list of two-dimensional functions).</p></li>
|
||||
<li><p>Compute and interpret a confusion matrix of your best classification model (see <a class="reference external" href="https://www.researchgate.net/figure/Confusion-matrix-of-MNIST-and-F-MNIST-embeddings_fig5_349758607">https://www.researchgate.net/figure/Confusion-matrix-of-MNIST-and-F-MNIST-embeddings_fig5_349758607</a>)</p></li>
|
||||
</ol>
|
||||
</section>
|
||||
</section>
|
||||
<section id="background-literature">
|
||||
<h2>Background literature<a class="headerlink" href="#background-literature" title="Link to this heading">#</a></h2>
|
||||
<ol class="arabic simple">
|
||||
@@ -807,6 +861,11 @@ we encourage you to collaborate. Optimal working groups consist of
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#part-g-critical-evaluation-of-the-various-algorithms">Part g) Critical evaluation of the various algorithms</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#summary-of-methods-to-implement-and-analyze">Summary of methods to implement and analyze</a><ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#required-analysis">Required Analysis:</a></li>
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#optional-note-that-you-should-include-at-least-two-of-these-in-the-report">Optional (Note that you should include at least two of these in the report):</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#background-literature">Background literature</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#introduction-to-numerical-projects">Introduction to numerical projects</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#format-for-electronic-delivery-of-report-and-programs">Format for electronic delivery of report and programs</a></li>
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -547,10 +547,8 @@ doconce format html week42.do.txt --no_mako -->
|
||||
<h2>Readings and videos<a class="headerlink" href="#readings-and-videos" title="Link to this heading">#</a></h2>
|
||||
<ol class="arabic simple">
|
||||
<li><p>These lecture notes</p></li>
|
||||
</ol>
|
||||
<!-- o [Video of lecture](https://youtu.be/7B2F35gNj2Y) -->
|
||||
<!-- o [Whiteboard notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOct14.pdf) -->
|
||||
<ol class="arabic simple" start="2">
|
||||
<li><p>Video of lecture at <a class="reference external" href="https://youtu.be/eqyNrEYRXnY">https://youtu.be/eqyNrEYRXnY</a></p></li>
|
||||
<li><p>Whiteboard notes at <a class="github reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2025/FYSSTKweek42.pdf">CompPhysics/MachineLearning</a></p></li>
|
||||
<li><p>For a more in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For the optimization part, see chapter 8.</p></li>
|
||||
<li><p>Neural Networks demystified at <a class="reference external" href="https://www.youtube.com/watch?v=bxe2T-V8XRs&amp;list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&amp;ab_channel=WelchLabs">https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs</a></p></li>
|
||||
<li><p>Building Neural Networks from scratch at <a class="reference external" href="https://www.youtube.com/watch?v=Wo5dMEP_BbI&amp;list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&amp;ab_channel=sentdex">https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex</a></p></li>
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "061af572",
|
||||
"id": "96e577ca",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "22578683",
|
||||
"id": "067c02b9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "61fb162f",
|
||||
"id": "01f9fedd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -58,7 +58,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "104c69e1",
|
||||
"id": "9f8e4871",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -104,7 +104,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0d2c42e3",
|
||||
"id": "460cc6ea",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -121,7 +121,7 @@
|
||||
"\n",
|
||||
"* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be\n",
|
||||
"\n",
|
||||
" * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
" * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
"\n",
|
||||
"* Classification.\n",
|
||||
"\n",
|
||||
@@ -132,7 +132,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8baab67",
|
||||
"id": "d62a07ef",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -162,7 +162,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "87e7ed71",
|
||||
"id": "9cd8b8ac",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -189,7 +189,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5a26b6ad",
|
||||
"id": "5931b155",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -205,7 +205,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "096fe6c4",
|
||||
"id": "b273fc8a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -217,7 +217,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd986596",
|
||||
"id": "e13db1ec",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -252,7 +252,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e853d4b6",
|
||||
"id": "4f864e31",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -270,7 +270,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fc2d413b",
|
||||
"id": "c9faeafd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -285,7 +285,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e6821051",
|
||||
"id": "d865c22b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -302,7 +302,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cba72d68",
|
||||
"id": "5270af8f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -328,7 +328,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "e16fb528",
|
||||
"id": "4e0e1fea",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -347,7 +347,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "73599f42",
|
||||
"id": "8fe85677",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -358,7 +358,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "f1a639ef",
|
||||
"id": "b28318b2",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -370,7 +370,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90fb7b41",
|
||||
"id": "97e02c71",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -381,7 +381,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "424af629",
|
||||
"id": "88af355c",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -394,7 +394,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3c006080",
|
||||
"id": "d1f8f0ed",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -407,7 +407,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a18ddd54",
|
||||
"id": "554b3a48",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -419,7 +419,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1a1afaf9",
|
||||
"id": "77bfdd5c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -434,7 +434,7 @@
|
||||
"code for classification and pertinent results against a similar code using **Scikit-Learn** or **tensorflow/keras** or **pytorch**.\n",
|
||||
"\n",
|
||||
"If you have time, you can use the functionality of **scikit-learn** and compare your neural network results with those from Logistic regression. This is optional.\n",
|
||||
"The weblink here <https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3>compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero. \n",
|
||||
"The weblink here <https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3>compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero and keeping just the input and the output layers. \n",
|
||||
"\n",
|
||||
"If you wish to compare with say Logisti Regression from **scikit-learn**, the following code uses the above data set"
|
||||
]
|
||||
@@ -442,7 +442,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "3c37cbaf",
|
||||
"id": "eaa9e72e",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -464,7 +464,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "106b9303",
|
||||
"id": "c7ba883e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -480,7 +480,88 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "55da0d7f",
|
||||
"id": "595be693",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Summary of methods to implement and analyze\n",
|
||||
"\n",
|
||||
"**Required Implementation:**\n",
|
||||
"1. Reuse the regression code and results from project 1, these will act as a benchmark for seeing how suited a neural network is for this regression task.\n",
|
||||
"\n",
|
||||
"2. Implement a neural network with\n",
|
||||
"\n",
|
||||
" * A flexible number of layers\n",
|
||||
"\n",
|
||||
" * A flexible number of nodes in each layer\n",
|
||||
"\n",
|
||||
" * A changeable activation function in each layer (Sigmoid, ReLU, LeakyReLU, as well as Linear and Softmax)\n",
|
||||
"\n",
|
||||
" * A changeable cost function, which will be set to MSE for regression and cross-entropy for multiple-classification\n",
|
||||
"\n",
|
||||
" * An optional L1 or L2 norm of the weights and biases in the cost function (only used for computing gradients, not interpretable metrics)\n",
|
||||
"\n",
|
||||
"3. Implement the back-propagation algorithm to compute the gradient of your neural network\n",
|
||||
"\n",
|
||||
"4. Reuse the implementation of Plain and Stochastic Gradient Descent from Project 1 (and adapt the code to work with the your neural network)\n",
|
||||
"\n",
|
||||
" * With no optimization algorithm\n",
|
||||
"\n",
|
||||
" * With RMS Prop\n",
|
||||
"\n",
|
||||
" * With ADAM\n",
|
||||
"\n",
|
||||
"5. Implement scaling and train-test splitting of your data, preferably using sklearn\n",
|
||||
"\n",
|
||||
"6. Implement and compute metrics like the MSE and Accuracy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "35138b41",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Required Analysis:\n",
|
||||
"\n",
|
||||
"1. Briefly show and argue for the advantages and disadvantages of the methods from Project 1.\n",
|
||||
"\n",
|
||||
"2. Explore and show the impact of changing the number of layers, nodes per layer, choice of activation function, and inclusion of L1 and L2 norms. Present only the most interesting results from this exploration. 2D Heatmaps will be good for this: Start with finding a well performing set of hyper-parameters, then change two at a time in a range that shows good and bad performance.\n",
|
||||
"\n",
|
||||
"3. Show and argue for the advantages and disadvantages of using a neural network for regression on your data\n",
|
||||
"\n",
|
||||
"4. Show and argue for the advantages and disadvantages of using a neural network for classification on your data\n",
|
||||
"\n",
|
||||
"5. Show and argue for the advantages and disadvantages of the different gradient methods and learning rates when training the neural network"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b18bea03",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Optional (Note that you should include at least two of these in the report):\n",
|
||||
"\n",
|
||||
"1. Implement Logistic Regression as simple classification model case (equivalent to a Neural Network with just the output layer)\n",
|
||||
"\n",
|
||||
"2. Compute the gradient of the neural network with autograd, to show that it gives the same result as your hand-written backpropagation.\n",
|
||||
"\n",
|
||||
"3. Compare your results with results from using a machine-learning library like pytorch (https://docs.pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html)\n",
|
||||
"\n",
|
||||
"4. Use a more complex classification dataset instead, like the fashion MNIST (see <https://www.kaggle.com/datasets/zalando-research/fashionmnist>)\n",
|
||||
"\n",
|
||||
"5. Use a more complex regression dataset instead, like the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
"\n",
|
||||
"6. Compute and interpret a confusion matrix of your best classification model (see <https://www.researchgate.net/figure/Confusion-matrix-of-MNIST-and-F-MNIST-embeddings_fig5_349758607>)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "580d8424",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -496,7 +577,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d3731e2c",
|
||||
"id": "96f5c67e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -527,7 +608,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6c7c5340",
|
||||
"id": "d1bc28ba",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+110
-29
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "061af572",
|
||||
"id": "96e577ca",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "22578683",
|
||||
"id": "067c02b9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "61fb162f",
|
||||
"id": "01f9fedd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -58,7 +58,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "104c69e1",
|
||||
"id": "9f8e4871",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -104,7 +104,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0d2c42e3",
|
||||
"id": "460cc6ea",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -121,7 +121,7 @@
|
||||
"\n",
|
||||
"* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be\n",
|
||||
"\n",
|
||||
" * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
" * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
"\n",
|
||||
"* Classification.\n",
|
||||
"\n",
|
||||
@@ -132,7 +132,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8baab67",
|
||||
"id": "d62a07ef",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -162,7 +162,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "87e7ed71",
|
||||
"id": "9cd8b8ac",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -189,7 +189,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5a26b6ad",
|
||||
"id": "5931b155",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -205,7 +205,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "096fe6c4",
|
||||
"id": "b273fc8a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -217,7 +217,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd986596",
|
||||
"id": "e13db1ec",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -252,7 +252,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e853d4b6",
|
||||
"id": "4f864e31",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -270,7 +270,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fc2d413b",
|
||||
"id": "c9faeafd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -285,7 +285,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e6821051",
|
||||
"id": "d865c22b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -302,7 +302,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cba72d68",
|
||||
"id": "5270af8f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -328,7 +328,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "e16fb528",
|
||||
"id": "4e0e1fea",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -347,7 +347,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "73599f42",
|
||||
"id": "8fe85677",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -358,7 +358,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "f1a639ef",
|
||||
"id": "b28318b2",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -370,7 +370,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90fb7b41",
|
||||
"id": "97e02c71",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -381,7 +381,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "424af629",
|
||||
"id": "88af355c",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -394,7 +394,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3c006080",
|
||||
"id": "d1f8f0ed",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -407,7 +407,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a18ddd54",
|
||||
"id": "554b3a48",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -419,7 +419,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1a1afaf9",
|
||||
"id": "77bfdd5c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -434,7 +434,7 @@
|
||||
"code for classification and pertinent results against a similar code using **Scikit-Learn** or **tensorflow/keras** or **pytorch**.\n",
|
||||
"\n",
|
||||
"If you have time, you can use the functionality of **scikit-learn** and compare your neural network results with those from Logistic regression. This is optional.\n",
|
||||
"The weblink here <https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3>compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero. \n",
|
||||
"The weblink here <https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3>compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero and keeping just the input and the output layers. \n",
|
||||
"\n",
|
||||
"If you wish to compare with say Logisti Regression from **scikit-learn**, the following code uses the above data set"
|
||||
]
|
||||
@@ -442,7 +442,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "3c37cbaf",
|
||||
"id": "eaa9e72e",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -464,7 +464,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "106b9303",
|
||||
"id": "c7ba883e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -480,7 +480,88 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "55da0d7f",
|
||||
"id": "595be693",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Summary of methods to implement and analyze\n",
|
||||
"\n",
|
||||
"**Required Implementation:**\n",
|
||||
"1. Reuse the regression code and results from project 1, these will act as a benchmark for seeing how suited a neural network is for this regression task.\n",
|
||||
"\n",
|
||||
"2. Implement a neural network with\n",
|
||||
"\n",
|
||||
" * A flexible number of layers\n",
|
||||
"\n",
|
||||
" * A flexible number of nodes in each layer\n",
|
||||
"\n",
|
||||
" * A changeable activation function in each layer (Sigmoid, ReLU, LeakyReLU, as well as Linear and Softmax)\n",
|
||||
"\n",
|
||||
" * A changeable cost function, which will be set to MSE for regression and cross-entropy for multiple-classification\n",
|
||||
"\n",
|
||||
" * An optional L1 or L2 norm of the weights and biases in the cost function (only used for computing gradients, not interpretable metrics)\n",
|
||||
"\n",
|
||||
"3. Implement the back-propagation algorithm to compute the gradient of your neural network\n",
|
||||
"\n",
|
||||
"4. Reuse the implementation of Plain and Stochastic Gradient Descent from Project 1 (and adapt the code to work with the your neural network)\n",
|
||||
"\n",
|
||||
" * With no optimization algorithm\n",
|
||||
"\n",
|
||||
" * With RMS Prop\n",
|
||||
"\n",
|
||||
" * With ADAM\n",
|
||||
"\n",
|
||||
"5. Implement scaling and train-test splitting of your data, preferably using sklearn\n",
|
||||
"\n",
|
||||
"6. Implement and compute metrics like the MSE and Accuracy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "35138b41",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Required Analysis:\n",
|
||||
"\n",
|
||||
"1. Briefly show and argue for the advantages and disadvantages of the methods from Project 1.\n",
|
||||
"\n",
|
||||
"2. Explore and show the impact of changing the number of layers, nodes per layer, choice of activation function, and inclusion of L1 and L2 norms. Present only the most interesting results from this exploration. 2D Heatmaps will be good for this: Start with finding a well performing set of hyper-parameters, then change two at a time in a range that shows good and bad performance.\n",
|
||||
"\n",
|
||||
"3. Show and argue for the advantages and disadvantages of using a neural network for regression on your data\n",
|
||||
"\n",
|
||||
"4. Show and argue for the advantages and disadvantages of using a neural network for classification on your data\n",
|
||||
"\n",
|
||||
"5. Show and argue for the advantages and disadvantages of the different gradient methods and learning rates when training the neural network"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b18bea03",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Optional (Note that you should include at least two of these in the report):\n",
|
||||
"\n",
|
||||
"1. Implement Logistic Regression as simple classification model case (equivalent to a Neural Network with just the output layer)\n",
|
||||
"\n",
|
||||
"2. Compute the gradient of the neural network with autograd, to show that it gives the same result as your hand-written backpropagation.\n",
|
||||
"\n",
|
||||
"3. Compare your results with results from using a machine-learning library like pytorch (https://docs.pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html)\n",
|
||||
"\n",
|
||||
"4. Use a more complex classification dataset instead, like the fashion MNIST (see <https://www.kaggle.com/datasets/zalando-research/fashionmnist>)\n",
|
||||
"\n",
|
||||
"5. Use a more complex regression dataset instead, like the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of <https://www.nature.com/articles/s41467-025-61362-4> for an extensive list of two-dimensional functions). \n",
|
||||
"\n",
|
||||
"6. Compute and interpret a confusion matrix of your best classification model (see <https://www.researchgate.net/figure/Confusion-matrix-of-MNIST-and-F-MNIST-embeddings_fig5_349758607>)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "580d8424",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -496,7 +577,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d3731e2c",
|
||||
"id": "96f5c67e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -527,7 +608,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6c7c5340",
|
||||
"id": "d1bc28ba",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user