update week 43

doc/LectureNotes/exercisesweek43.ipynb

@@ -3,9 +3,7 @@
   {
    "cell_type": "markdown",
    "id": "ae832182",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
     "doconce format html exercisesweek43.do.txt  -->\n",
@@ -15,9 +13,7 @@
   {
    "cell_type": "markdown",
    "id": "77f844d9",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "# Exercises weeks 43 and 44 \n",
     "**October 23-27, 2023**\n",
@@ -30,9 +26,7 @@
   {
    "cell_type": "markdown",
    "id": "d5b983e3",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "# Overarching aims of the exercises weeks 43 and 44\n",
     "\n",
@@ -70,9 +64,7 @@
   {
    "cell_type": "markdown",
    "id": "7f6cc12f",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## The AND and XOR Gates\n",
     "\n",
@@ -108,9 +100,7 @@
   {
    "cell_type": "markdown",
    "id": "c0b07e46",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Representing the Data Sets\n",
     "\n",
@@ -120,9 +110,7 @@
   {
    "cell_type": "markdown",
    "id": "e0a9840a",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -135,9 +123,7 @@
   {
    "cell_type": "markdown",
    "id": "ff95d379",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
     "\n",
@@ -165,9 +151,7 @@
   {
    "cell_type": "markdown",
    "id": "36fa3466",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Setting up dimensionalities by hand\n",
     "\n",
@@ -177,9 +161,7 @@
   {
    "cell_type": "markdown",
    "id": "e2e5e808",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n",
@@ -190,9 +172,7 @@
   {
    "cell_type": "markdown",
    "id": "419e3904",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives"
    ]
@@ -200,9 +180,7 @@
   {
    "cell_type": "markdown",
    "id": "88a9712a",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -215,9 +193,7 @@
   {
    "cell_type": "markdown",
    "id": "036eee14",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Assume also that the bias vector for the hidden layer is"
    ]
@@ -225,9 +201,7 @@
   {
    "cell_type": "markdown",
    "id": "3c771918",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n",
@@ -238,9 +212,7 @@
   {
    "cell_type": "markdown",
    "id": "f5a4150d",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Adding it gives us the input to the activation function of the hidden layer"
    ]
@@ -248,9 +220,7 @@
   {
    "cell_type": "markdown",
    "id": "e4eb7753",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n",
@@ -263,9 +233,7 @@
   {
    "cell_type": "markdown",
    "id": "b5442b32",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have"
    ]
@@ -273,9 +241,7 @@
   {
    "cell_type": "markdown",
    "id": "64d99f85",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -288,9 +254,7 @@
   {
    "cell_type": "markdown",
    "id": "2aee4b6b",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Assume also that the bias of the output layer is zero and that the weights of the output layer are"
    ]
@@ -298,9 +262,7 @@
   {
    "cell_type": "markdown",
    "id": "884541eb",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n",
@@ -311,9 +273,7 @@
   {
    "cell_type": "markdown",
    "id": "3999a4d5",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "and multiplying with $\\boldsymbol{a}_h$ gives the output"
    ]
@@ -321,9 +281,7 @@
   {
    "cell_type": "markdown",
    "id": "a4896b4d",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "$$\n",
     "\\boldsymbol{a}_o=\\boldsymbol{w}_h^T\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -336,9 +294,7 @@
   {
    "cell_type": "markdown",
    "id": "2c74e7c0",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "the wanted result."
    ]
@@ -346,9 +302,7 @@
   {
    "cell_type": "markdown",
    "id": "249b9804",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Setting up the Neural Network\n",
     "\n",
@@ -359,10 +313,7 @@
    "cell_type": "code",
    "execution_count": 1,
    "id": "afdd3100",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "%matplotlib inline\n",
@@ -429,9 +380,7 @@
   {
    "cell_type": "markdown",
    "id": "fb383164",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
    ]
@@ -439,9 +388,7 @@
   {
    "cell_type": "markdown",
    "id": "0d26394c",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## The Code using Scikit-Learn"
    ]
@@ -450,10 +397,7 @@
    "cell_type": "code",
    "execution_count": 2,
    "id": "befc368c",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# import necessary packages\n",
@@ -515,9 +459,7 @@
   {
    "cell_type": "markdown",
    "id": "03895a9f",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Building a neural network code\n",
     "\n",
@@ -534,9 +476,7 @@
   {
    "cell_type": "markdown",
    "id": "6226915b",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Learning rate methods\n",
     "\n",
@@ -556,10 +496,7 @@
    "cell_type": "code",
    "execution_count": 3,
    "id": "b6d5ea73",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "import autograd.numpy as np\n",
@@ -697,9 +634,7 @@
   {
    "cell_type": "markdown",
    "id": "edb42bef",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Usage of the above learning rate schedulers\n",
     "\n",
@@ -713,10 +648,7 @@
    "cell_type": "code",
    "execution_count": 4,
    "id": "108c1209",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
@@ -726,9 +658,7 @@
   {
    "cell_type": "markdown",
    "id": "d1c423f0",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Here is a small example for how a segment of code using schedulers\n",
     "could look. Switching out the schedulers is simple."
@@ -738,10 +668,7 @@
    "cell_type": "code",
    "execution_count": 5,
    "id": "63c8c749",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "weights = np.ones((3,3))\n",
@@ -760,9 +687,7 @@
   {
    "cell_type": "markdown",
    "id": "7cb80992",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Cost functions\n",
     "\n",
@@ -776,10 +701,7 @@
    "cell_type": "code",
    "execution_count": 6,
    "id": "a181dacb",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "import autograd.numpy as np\n",
@@ -814,9 +736,7 @@
   {
    "cell_type": "markdown",
    "id": "26a7fe37",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Below we give a short example of how these cost function may be used\n",
     "to obtain results if you wish to test them out on your own using\n",
@@ -827,10 +747,7 @@
    "cell_type": "code",
    "execution_count": 7,
    "id": "5894bd31",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "from autograd import grad\n",
@@ -848,9 +765,7 @@
   {
    "cell_type": "markdown",
    "id": "71141fab",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Activation functions\n",
     "\n",
@@ -864,10 +779,7 @@
    "cell_type": "code",
    "execution_count": 8,
    "id": "433e0886",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "import autograd.numpy as np\n",
@@ -922,9 +834,7 @@
   {
    "cell_type": "markdown",
    "id": "14c9d538",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Below follows a short demonstration of how to use an activation\n",
     "function. The derivative of the activation function will be important\n",
@@ -937,10 +847,7 @@
    "cell_type": "code",
    "execution_count": 9,
    "id": "4c3a44ef",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "z = np.array([[4, 5, 6]]).T\n",
@@ -958,9 +865,7 @@
   {
    "cell_type": "markdown",
    "id": "aa7f604e",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### The Neural Network\n",
     "\n",
@@ -982,10 +887,7 @@
    "cell_type": "code",
    "execution_count": 10,
    "id": "585a227b",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "import math\n",
@@ -1454,9 +1356,7 @@
   {
    "cell_type": "markdown",
    "id": "7de704af",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Before we make a model, we will quickly generate a dataset we can use\n",
     "for our linear regression problem as shown below"
@@ -1466,10 +1366,7 @@
    "cell_type": "code",
    "execution_count": 11,
    "id": "212b0a35",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "import autograd.numpy as np\n",
@@ -1510,9 +1407,7 @@
   {
    "cell_type": "markdown",
    "id": "1ff4ec7b",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Now that we have our dataset ready for the regression, we can create\n",
     "our regressor. Note that with the seed parameter, we can make sure our\n",
@@ -1526,10 +1421,7 @@
    "cell_type": "code",
    "execution_count": 12,
    "id": "57d8eca3",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "input_nodes = X_train.shape[1]\n",
@@ -1541,9 +1433,7 @@
   {
    "cell_type": "markdown",
    "id": "f1977bae",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "We then fit our model with our training data using the scheduler of our choice."
    ]
@@ -1552,10 +1442,7 @@
    "cell_type": "code",
    "execution_count": 13,
    "id": "31ae2a1e",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1567,9 +1454,7 @@
   {
    "cell_type": "markdown",
    "id": "96a64da4",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Due to the progress bar we can see the MSE (train_error) throughout\n",
     "the FFNN's training. Note that the fit() function has some optional\n",
@@ -1583,10 +1468,7 @@
    "cell_type": "code",
    "execution_count": 14,
    "id": "246499fc",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1597,9 +1479,7 @@
   {
    "cell_type": "markdown",
    "id": "61d11e17",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "We see that given more epochs to train on, the regressor reaches a lower MSE.\n",
     "\n",
@@ -1612,10 +1492,7 @@
    "cell_type": "code",
    "execution_count": 15,
    "id": "b5b91899",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "from sklearn.datasets import load_breast_cancer\n",
@@ -1638,10 +1515,7 @@
    "cell_type": "code",
    "execution_count": 16,
    "id": "4bd01b3d",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "input_nodes = X_train.shape[1]\n",
@@ -1653,9 +1527,7 @@
   {
    "cell_type": "markdown",
    "id": "3a138d66",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "We will now make use of our validation data by passing it into our fit function as a keyword argument"
    ]
@@ -1664,10 +1536,7 @@
    "cell_type": "code",
    "execution_count": 17,
    "id": "faf1534e",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1679,9 +1548,7 @@
   {
    "cell_type": "markdown",
    "id": "33d2d346",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Finally, we will create a neural network with 2 hidden layers with activation functions."
    ]
@@ -1690,10 +1557,7 @@
    "cell_type": "code",
    "execution_count": 18,
    "id": "a8f0a61c",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "input_nodes = X_train.shape[1]\n",
@@ -1710,10 +1574,7 @@
    "cell_type": "code",
    "execution_count": 19,
    "id": "d651702d",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
@@ -1725,9 +1586,7 @@
   {
    "cell_type": "markdown",
    "id": "200b64c0",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "### Multiclass classification\n",
     "\n",
@@ -1740,10 +1599,7 @@
    "cell_type": "code",
    "execution_count": 20,
    "id": "98b8d59b",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "from sklearn.datasets import load_digits\n",
@@ -1777,9 +1633,7 @@
   {
    "cell_type": "markdown",
    "id": "cadd7e47",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "## Testing the XOR gate and other gates\n",
     "\n",
@@ -1790,10 +1644,7 @@
    "cell_type": "code",
    "execution_count": 21,
    "id": "a40e5cfe",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
@@ -1813,15 +1664,31 @@
   {
    "cell_type": "markdown",
    "id": "b635db73",
-   "metadata": {
-    "editable": true
-   },
+   "metadata": {},
    "source": [
     "Not bad, but the results depend strongly on the learning reate. Try different learning rates."
    ]
   }
  ],
- "metadata": {},
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.10"
+  }
+ },
  "nbformat": 4,
  "nbformat_minor": 5
 }