diff --git a/doc/LectureNotes/.ipynb_checkpoints/exercisesweek43-checkpoint.ipynb b/doc/LectureNotes/.ipynb_checkpoints/exercisesweek43-checkpoint.ipynb
index 6d6019289..f80e8787a 100644
--- a/doc/LectureNotes/.ipynb_checkpoints/exercisesweek43-checkpoint.ipynb
+++ b/doc/LectureNotes/.ipynb_checkpoints/exercisesweek43-checkpoint.ipynb
@@ -2,1492 +2,624 @@
"cells": [
{
"cell_type": "markdown",
- "id": "b2937d10",
- "metadata": {},
+ "id": "860d70d8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
- ""
+ ""
]
},
{
"cell_type": "markdown",
- "id": "3dd00d19",
- "metadata": {},
+ "id": "119c0988",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "# Exercises weeks 43 and 44 \n",
- "**October 23-27, 2023**\n",
+ "# Exercises week 43 \n",
+ "**October 20-24, 2025**\n",
"\n",
- "Date: **Deadline is Sunday November 5 at midnight**\n",
- "\n",
- "You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points."
+ "Date: **Deadline Friday October 24 at midnight**"
]
},
{
"cell_type": "markdown",
- "id": "82a19a1d",
- "metadata": {},
+ "id": "909887eb",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "# Overarching aims of the exercises weeks 43 and 44\n",
+ "# Overarching aims of the exercises for week 43\n",
"\n",
- "The aim of the exercises this week and next week is to get started with writing a neural network code\n",
- "of relevance for project 2. \n",
+ "The aim of the exercises this week is to gain some confidence with\n",
+ "ways to visualize the results of a classification problem. We will\n",
+ "target three ways of setting up the analysis. The first and simplest\n",
+ "one is the\n",
+ "1. so-called confusion matrix. The next one is the so-called\n",
"\n",
- "During week 41 we discussed three different types of gates, the\n",
- "so-called XOR, the OR and the AND gates. In order to develop a code\n",
- "for neural networks, it can be useful to set up a simpler system with\n",
- "only two inputs and one output. This can make it easier to debug and\n",
- "study the feed forward pass and the back propagation part. In the\n",
- "exercise this and next week, we propose to study this system with just\n",
- "one hidden layer and two hidden nodes. There is only one output node\n",
- "and we can choose to use either a simple regression case (fitting a\n",
- "line) or just a binary classification case with the cross-entropy as\n",
- "cost function.\n",
+ "2. ROC curve. Finally we have the\n",
"\n",
- "Their inputs and outputs can be\n",
- "summarized using the following tables, first for the OR gate with\n",
- "inputs $x_1$ and $x_2$ and outputs $y$:\n",
+ "3. Cumulative gain curve.\n",
"\n",
- "\n",
- "\n",
- "| $x_1$ | $x_2$ | $y$ |
|---|
\n",
- "\n",
- "\n",
- "| 0 | 0 | 0 |
\n",
- "| 0 | 1 | 1 |
\n",
- "| 1 | 0 | 1 |
\n",
- "| 1 | 1 | 1 |
\n",
- "\n",
- "
"
+ "We will use Logistic Regression as method for the classification in\n",
+ "this exercise. You can compare these results with those obtained with\n",
+ "your neural network code from project 2 without a hidden layer.\n",
+ "\n",
+ "In these exercises we will use binary and multi-class data sets\n",
+ "(the Iris data set from week 41).\n",
+ "\n",
+ "The underlying mathematics is described here."
]
},
{
"cell_type": "markdown",
- "id": "f74f69af",
- "metadata": {},
+ "id": "1e1cb4fb",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "## The AND and XOR Gates\n",
+ "### Confusion Matrix\n",
"\n",
- "The AND gate is defined as\n",
- "\n",
- "\n",
- "\n",
- "| $x_1$ | $x_2$ | $y$ |
|---|
\n",
- "\n",
- "\n",
- "| 0 | 0 | 0 |
\n",
- "| 0 | 1 | 0 |
\n",
- "| 1 | 0 | 0 |
\n",
- "| 1 | 1 | 1 |
\n",
- "\n",
- "
\n",
- "\n",
- "And finally we have the XOR gate\n",
- "\n",
- "\n",
- "\n",
- "| $x_1$ | $x_2$ | $y$ |
|---|
\n",
- "\n",
- "\n",
- "| 0 | 0 | 0 |
\n",
- "| 0 | 1 | 1 |
\n",
- "| 1 | 0 | 1 |
\n",
- "| 1 | 1 | 0 |
\n",
- "\n",
- "
"
+ "A **confusion matrix** summarizes a classifier’s performance by\n",
+ "tabulating predictions versus true labels. For binary classification,\n",
+ "it is a $2\\times2$ table whose entries are counts of outcomes:"
]
},
{
"cell_type": "markdown",
- "id": "1b52d47a",
- "metadata": {},
- "source": [
- "## Representing the Data Sets\n",
- "\n",
- "Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3e6910cb",
- "metadata": {},
+ "id": "7b090385",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
- "\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
- " 0 & 1 \\\\\n",
- "\t\t 1 & 0 \\\\\n",
- "\t\t 1 & 1 \\end{bmatrix},\n",
+ "\\begin{array}{l|cc} & \\text{Predicted Positive} & \\text{Predicted Negative} \\\\ \\hline \\text{Actual Positive} & TP & FN \\\\ \\text{Actual Negative} & FP & TN \\end{array}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "90a3b78a",
- "metadata": {},
+ "id": "1e14904b",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
- "\n",
- "Your tasks here are\n",
- "\n",
- "1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.\n",
- "\n",
- "2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.\n",
- "\n",
- "3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.\n",
- "\n",
- "4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.\n",
- "\n",
- "5. Set up the cost function (cross entropy for classification of binary cases).\n",
- "\n",
- "6. Calculate the gradients needed for the back propagation part.\n",
- "\n",
- "7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.\n",
- "\n",
- "8. Train the network and study your results and compare with results obtained either with **scikit-learn** or **TensorFlow**.\n",
- "\n",
- "Everything you develop here can be used directly into the code for the project."
+ "Here TP (true positives) is the number of cases correctly predicted as\n",
+ "positive, FP (false positives) is the number incorrectly predicted as\n",
+ "positive, TN (true negatives) is correctly predicted negative, and FN\n",
+ "(false negatives) is incorrectly predicted negative . In other words,\n",
+ "“positive” means class 1 and “negative” means class 0; for example, TP\n",
+ "occurs when the prediction and actual are both positive. Formally:"
]
},
{
"cell_type": "markdown",
- "id": "d6a3ab1e",
- "metadata": {},
+ "id": "e93ea290",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "## Setting up the Neural Network\n",
+ "$$\n",
+ "\\text{TPR} = \\frac{\\text{TP}}{\\text{TP} + \\text{FN}}, \\quad \\text{FPR} = \\frac{\\text{FP}}{\\text{FP} + \\text{TN}},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c80bea5b",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "where TPR and FPR are the true and false positive rates defined below.\n",
"\n",
- "We define first our design matrix and the various output vectors for the different gates."
+ "In multiclass classification with $K$ classes, the confusion matrix\n",
+ "generalizes to a $K\\times K$ table. Entry $N_{ij}$ in the table is\n",
+ "the count of instances whose true class is $i$ and whose predicted\n",
+ "class is $j$. For example, a three-class confusion matrix can be written\n",
+ "as:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a0f68f5f",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\begin{array}{c|ccc} & \\text{Pred Class 1} & \\text{Pred Class 2} & \\text{Pred Class 3} \\\\ \\hline \\text{Act Class 1} & N_{11} & N_{12} & N_{13} \\\\ \\text{Act Class 2} & N_{21} & N_{22} & N_{23} \\\\ \\text{Act Class 3} & N_{31} & N_{32} & N_{33} \\end{array}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "869669b2",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "Here the diagonal entries $N_{ii}$ are the true positives for each\n",
+ "class, and off-diagonal entries are misclassifications. This matrix\n",
+ "allows computation of per-class metrics: e.g. for class $i$,\n",
+ "$\\mathrm{TP}_i=N_{ii}$, $\\mathrm{FN}_i=\\sum_{j\\neq i}N_{ij}$,\n",
+ "$\\mathrm{FP}_i=\\sum_{j\\neq i}N_{ji}$, and $\\mathrm{TN}_i$ is the sum of\n",
+ "all remaining entries.\n",
+ "\n",
+ "As defined above, TPR and FPR come from the binary case. In binary\n",
+ "terms with $P$ actual positives and $N$ actual negatives, one has"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2abd82a7",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\text{TPR} = \\frac{TP}{P} = \\frac{TP}{TP+FN}, \\quad \\text{FPR} =\n",
+ "\\frac{FP}{N} = \\frac{FP}{FP+TN},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2f79325c",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "as used in standard confusion-matrix\n",
+ "formulations. These rates will be used in constructing ROC curves."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0ce65a47",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "### ROC Curve\n",
+ "\n",
+ "The Receiver Operating Characteristic (ROC) curve plots the trade-off\n",
+ "between true positives and false positives as a discrimination\n",
+ "threshold varies. Specifically, for a binary classifier that outputs\n",
+ "a score or probability, one varies the threshold $t$ for declaring\n",
+ "**positive**, and computes at each $t$ the true positive rate\n",
+ "$\\mathrm{TPR}(t)$ and false positive rate $\\mathrm{FPR}(t)$ using the\n",
+ "confusion matrix at that threshold. The ROC curve is then the graph\n",
+ "of TPR versus FPR. By definition,"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d750fdff",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\mathrm{TPR} = \\frac{TP}{TP+FN}, \\qquad \\mathrm{FPR} = \\frac{FP}{FP+TN},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "561bfb2c",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "where $TP,FP,TN,FN$ are counts determined by threshold $t$. A perfect\n",
+ "classifier would reach the point (FPR=0, TPR=1) at some threshold.\n",
+ "\n",
+ "Formally, the ROC curve is obtained by plotting\n",
+ "$(\\mathrm{FPR}(t),\\mathrm{TPR}(t))$ for all $t\\in[0,1]$ (or as $t$\n",
+ "sweeps through the sorted scores). The Area Under the ROC Curve (AUC)\n",
+ "quantifies the average performance over all thresholds. It can be\n",
+ "interpreted probabilistically: $\\mathrm{AUC} =\n",
+ "\\Pr\\bigl(s(X^+)>s(X^-)\\bigr)$, the probability that a random positive\n",
+ "instance $X^+$ receives a higher score $s$ than a random negative\n",
+ "instance $X^-$ . Equivalently, the AUC is the integral under the ROC\n",
+ "curve:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5ca722fe",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\mathrm{AUC} \\;=\\; \\int_{0}^{1} \\mathrm{TPR}(f)\\,df,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "30080a86",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "where $f$ ranges over FPR (or fraction of negatives). A model that guesses at random yields a diagonal ROC (AUC=0.5), whereas a perfect model yields AUC=1.0."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9e627156",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "### Cumulative Gain\n",
+ "\n",
+ "The cumulative gain curve (or gains chart) evaluates how many\n",
+ "positives are captured as one targets an increasing fraction of the\n",
+ "population, sorted by model confidence. To construct it, sort all\n",
+ "instances by decreasing predicted probability of the positive class.\n",
+ "Then, for the top $\\alpha$ fraction of instances, compute the fraction\n",
+ "of all actual positives that fall in this subset. In formula form, if\n",
+ "$P$ is the total number of positive instances and $P(\\alpha)$ is the\n",
+ "number of positives among the top $\\alpha$ of the data, the cumulative\n",
+ "gain at level $\\alpha$ is"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3e9132ef",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\mathrm{Gain}(\\alpha) \\;=\\; \\frac{P(\\alpha)}{P}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "75be6f5c",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "For example, cutting off at the top 10% of predictions yields a gain\n",
+ "equal to (positives in top 10%) divided by (total positives) .\n",
+ "Plotting $\\mathrm{Gain}(\\alpha)$ versus $\\alpha$ (often in percent)\n",
+ "gives the gain curve. The baseline (random) curve is the diagonal\n",
+ "$\\mathrm{Gain}(\\alpha)=\\alpha$, while an ideal model has a steep climb\n",
+ "toward 1.\n",
+ "\n",
+ "A related measure is the {\\em lift}, often called the gain ratio. It is the ratio of the model’s capture rate to that of random selection. Equivalently,"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5525570",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\mathrm{Lift}(\\alpha) \\;=\\; \\frac{\\mathrm{Gain}(\\alpha)}{\\alpha}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "18ff8dc2",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "A lift $>1$ indicates better-than-random targeting. In practice, gain\n",
+ "and lift charts (used e.g.\\ in marketing or imbalanced classification)\n",
+ "show how many positives can be “gained” by focusing on a fraction of\n",
+ "the population ."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c3d3fde8",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "### Other measures: Precision, Recall, and the F$_1$ Measure\n",
+ "\n",
+ "Precision and recall (sensitivity) quantify binary classification\n",
+ "accuracy in terms of positive predictions. They are defined from the\n",
+ "confusion matrix as:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f1f14c8e",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\text{Precision} = \\frac{TP}{TP + FP}, \\qquad \\text{Recall} = \\frac{TP}{TP + FN}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "422cc743",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "Precision is the fraction of predicted positives that are correct, and\n",
+ "recall is the fraction of actual positives that are correctly\n",
+ "identified . A high-precision classifier makes few false-positive\n",
+ "errors, while a high-recall classifier makes few false-negative\n",
+ "errors.\n",
+ "\n",
+ "The F$_1$ score (balanced F-measure) combines precision and recall into a single metric via their harmonic mean. The usual formula is:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "621a2e8b",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "F_1 =2\\frac{\\text{Precision}\\times\\text{Recall}}{\\text{Precision} + \\text{Recall}}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "62eee54a",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "This can be shown to equal"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7a6a2e7a",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\frac{2\\,TP}{2\\,TP + FP + FN}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b96c9ff4",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "The F$_1$ score ranges from 0 (worst) to 1 (best), and balances the\n",
+ "trade-off between precision and recall.\n",
+ "\n",
+ "For multi-class classification, one computes per-class\n",
+ "precision/recall/F$_1$ (treating each class as “positive” in a\n",
+ "one-vs-rest manner) and then averages. Common averaging methods are:\n",
+ "\n",
+ "Micro-averaging: Sum all true positives, false positives, and false negatives across classes, then compute precision/recall/F$_1$ from these totals.\n",
+ "Macro-averaging: Compute the F$1$ score $F{1,i}$ for each class $i$ separately, then take the unweighted mean: $F_{1,\\mathrm{macro}} = \\frac{1}{K}\\sum_{i=1}^K F_{1,i}$ . This treats all classes equally regardless of size.\n",
+ "Weighted-averaging: Like macro-average, but weight each class’s $F_{1,i}$ by its support $n_i$ (true count): $F_{1,\\mathrm{weighted}} = \\frac{1}{N}\\sum_{i=1}^K n_i F_{1,i}$, where $N=\\sum_i n_i$. This accounts for class imbalance by giving more weight to larger classes .\n",
+ "\n",
+ "Each of these averages has different use-cases. Micro-average is\n",
+ "dominated by common classes, macro-average highlights performance on\n",
+ "rare classes, and weighted-average is a compromise. These formulas\n",
+ "and concepts allow rigorous evaluation of classifier performance in\n",
+ "both binary and multi-class settings."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9274bf3f",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Exercises\n",
+ "\n",
+ "Here is a simple code example which uses the Logistic regression machinery from **scikit-learn**.\n",
+ "At the end it sets up the confusion matrix and the ROC and cumulative gain curves.\n",
+ "Feel free to use these functionalities (we don't expect you to write your own code for say the confusion matrix)."
]
},
{
"cell_type": "code",
"execution_count": 1,
- "id": "152123b0",
- "metadata": {},
+ "id": "be9ff0b9",
+ "metadata": {
+ "collapsed": false,
+ "editable": true,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
- "\"\"\"\n",
- "Simple code that tests XOR, OR and AND gates with linear regression\n",
- "\"\"\"\n",
- "\n",
- "# import necessary packages\n",
- "import numpy as np\n",
"import matplotlib.pyplot as plt\n",
- "from sklearn import datasets\n",
- "\n",
- "def sigmoid(x):\n",
- " return 1/(1 + np.exp(-x))\n",
- "\n",
- "def feed_forward(X):\n",
- " # weighted sum of inputs to the hidden layer\n",
- " z_h = np.matmul(X, hidden_weights) + hidden_bias\n",
- " # activation in the hidden layer\n",
- " a_h = sigmoid(z_h)\n",
- " \n",
- " # weighted sum of inputs to the output layer\n",
- " z_o = np.matmul(a_h, output_weights) + output_bias\n",
- " # softmax output\n",
- " # axis 0 holds each input and axis 1 the probabilities of each category\n",
- " probabilities = sigmoid(z_o)\n",
- " return probabilities\n",
- "\n",
- "# we obtain a prediction by taking the class with the highest likelihood\n",
- "def predict(X):\n",
- " probabilities = feed_forward(X)\n",
- " return np.argmax(probabilities, axis=1)\n",
- "\n",
- "# ensure the same random numbers appear every time\n",
- "np.random.seed(0)\n",
- "\n",
- "# Design matrix\n",
- "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
- "\n",
- "# The XOR gate\n",
- "yXOR = np.array( [ 0, 1 ,1, 0])\n",
- "# The OR gate\n",
- "yOR = np.array( [ 0, 1 ,1, 1])\n",
- "# The AND gate\n",
- "yAND = np.array( [ 0, 0 ,0, 1])\n",
- "\n",
- "# Defining the neural network\n",
- "n_inputs, n_features = X.shape\n",
- "n_hidden_neurons = 2\n",
- "n_categories = 2\n",
- "n_features = 2\n",
- "\n",
- "# we make the weights normally distributed using numpy.random.randn\n",
- "\n",
- "# weights and bias in the hidden layer\n",
- "hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n",
- "hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n",
- "\n",
- "# weights and bias in the output layer\n",
- "output_weights = np.random.randn(n_hidden_neurons, n_categories)\n",
- "output_bias = np.zeros(n_categories) + 0.01\n",
- "\n",
- "probabilities = feed_forward(X)\n",
- "print(probabilities)\n",
- "\n",
- "\n",
- "predictions = predict(X)\n",
- "print(predictions)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "73319f0a",
- "metadata": {},
- "source": [
- "Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a7e0c47a",
- "metadata": {},
- "source": [
- "## The Code using Scikit-Learn"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "dbbacc67",
- "metadata": {},
- "outputs": [],
- "source": [
- "# import necessary packages\n",
"import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from sklearn.neural_network import MLPClassifier\n",
- "from sklearn.metrics import accuracy_score\n",
- "import seaborn as sns\n",
+ "from sklearn.model_selection import train_test_split \n",
+ "# from sklearn.datasets import fill in the data set\n",
+ "from sklearn.linear_model import LogisticRegression\n",
"\n",
- "# ensure the same random numbers appear every time\n",
- "np.random.seed(0)\n",
+ "# Load the data, fill inn\n",
+ "mydata.data = ?\n",
"\n",
- "# Design matrix\n",
- "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
+ "X_train, X_test, y_train, y_test = train_test_split(mydata.data,cancer.target,random_state=0)\n",
+ "print(X_train.shape)\n",
+ "print(X_test.shape)\n",
+ "# Logistic Regression\n",
+ "# define which type of problem, binary or multiclass\n",
+ "logreg = LogisticRegression(solver='lbfgs')\n",
+ "logreg.fit(X_train, y_train)\n",
"\n",
- "# The XOR gate\n",
- "yXOR = np.array( [ 0, 1 ,1, 0])\n",
- "# The OR gate\n",
- "yOR = np.array( [ 0, 1 ,1, 1])\n",
- "# The AND gate\n",
- "yAND = np.array( [ 0, 0 ,0, 1])\n",
+ "from sklearn.preprocessing import LabelEncoder\n",
+ "from sklearn.model_selection import cross_validate\n",
+ "#Cross validation\n",
+ "accuracy = cross_validate(logreg,X_test,y_test,cv=10)['test_score']\n",
+ "print(accuracy)\n",
+ "print(\"Test set accuracy with Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
"\n",
- "# Defining the neural network\n",
- "n_inputs, n_features = X.shape\n",
- "n_hidden_neurons = 2\n",
- "n_categories = 2\n",
- "n_features = 2\n",
- "\n",
- "eta_vals = np.logspace(-5, 1, 7)\n",
- "lmbd_vals = np.logspace(-5, 1, 7)\n",
- "# store models for later use\n",
- "DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n",
- "epochs = 100\n",
- "\n",
- "for i, eta in enumerate(eta_vals):\n",
- " for j, lmbd in enumerate(lmbd_vals):\n",
- " dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n",
- " alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n",
- " dnn.fit(X, yXOR)\n",
- " DNN_scikit[i][j] = dnn\n",
- " print(\"Learning rate = \", eta)\n",
- " print(\"Lambda = \", lmbd)\n",
- " print(\"Accuracy score on data set: \", dnn.score(X, yXOR))\n",
- " print()\n",
- "\n",
- "sns.set()\n",
- "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n",
- "for i in range(len(eta_vals)):\n",
- " for j in range(len(lmbd_vals)):\n",
- " dnn = DNN_scikit[i][j]\n",
- " test_pred = dnn.predict(X)\n",
- " test_accuracy[i][j] = accuracy_score(yXOR, test_pred)\n",
- "\n",
- "fig, ax = plt.subplots(figsize = (10, 10))\n",
- "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n",
- "ax.set_title(\"Test Accuracy\")\n",
- "ax.set_ylabel(\"$\\eta$\")\n",
- "ax.set_xlabel(\"$\\lambda$\")\n",
+ "import scikitplot as skplt\n",
+ "y_pred = logreg.predict(X_test)\n",
+ "skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)\n",
+ "plt.show()\n",
+ "y_probas = logreg.predict_proba(X_test)\n",
+ "skplt.metrics.plot_roc(y_test, y_probas)\n",
+ "plt.show()\n",
+ "skplt.metrics.plot_cumulative_gain(y_test, y_probas)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
- "id": "1cac501a",
- "metadata": {},
+ "id": "51760b3e",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "## Building a neural network code\n",
+ "### Exercise a)\n",
"\n",
- "Here we present a flexible object oriented codebase\n",
- "for a feed forward neural network, along with a demonstration of how\n",
- "to use it. Before we get into the details of the neural network, we\n",
- "will first present some implementations of various schedulers, cost\n",
- "functions and activation functions that can be used together with the\n",
- "neural network.\n",
- "\n",
- "The codes here were developed by Eric Reber and Gregor Kajda during spring 2023."
+ "Convince yourself about the mathematics for the confusion matrix, the ROC and the cumlative gain curves for both a binary and a multiclass classification problem."
]
},
{
"cell_type": "markdown",
- "id": "dd153528",
- "metadata": {},
+ "id": "c1d42f5f",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "### Learning rate methods\n",
+ "### Exercise b)\n",
"\n",
- "The code below shows object oriented implementations of the Constant,\n",
- "Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All\n",
- "of the classes belong to the shared abstract Scheduler class, and\n",
- "share the update_change() and reset() methods allowing for any of the\n",
- "schedulers to be seamlessly used during the training stage, as will\n",
- "later be shown in the fit() method of the neural\n",
- "network. Update_change() only has one parameter, the gradient\n",
- "($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted\n",
- "from the weights. The reset() function takes no parameters, and resets\n",
- "the desired variables. For Constant and Momentum, reset does nothing."
+ "Use a binary classification data available from **scikit-learn**. As an example you can use\n",
+ "the MNIST data set and just specialize to two numbers. To do so you can use the following code lines"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "d20bb8be",
+ "metadata": {
+ "collapsed": false,
+ "editable": true,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
+ "outputs": [],
+ "source": [
+ "from sklearn.datasets import load_digits\n",
+ "digits = load_digits(n_class=2) # Load only two classes, e.g., 0 and 1\n",
+ "X, y = digits.data, digits.target"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "828ea1cd",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "Alternatively, you can use the _make$\\_$classification_\n",
+ "functionality. This function generates a random $n$-class classification\n",
+ "dataset, which can be configured for binary classification by setting\n",
+ "n_classes=2. You can also control the number of samples, features,\n",
+ "informative features, redundant features, and more."
]
},
{
"cell_type": "code",
"execution_count": 3,
- "id": "f55eea63",
- "metadata": {},
+ "id": "d271f0ba",
+ "metadata": {
+ "collapsed": false,
+ "editable": true,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
- "import autograd.numpy as np\n",
- "\n",
- "class Scheduler:\n",
- " \"\"\"\n",
- " Abstract class for Schedulers\n",
- " \"\"\"\n",
- "\n",
- " def __init__(self, eta):\n",
- " self.eta = eta\n",
- "\n",
- " # should be overwritten\n",
- " def update_change(self, gradient):\n",
- " raise NotImplementedError\n",
- "\n",
- " # overwritten if needed\n",
- " def reset(self):\n",
- " pass\n",
- "\n",
- "\n",
- "class Constant(Scheduler):\n",
- " def __init__(self, eta):\n",
- " super().__init__(eta)\n",
- "\n",
- " def update_change(self, gradient):\n",
- " return self.eta * gradient\n",
- " \n",
- " def reset(self):\n",
- " pass\n",
- "\n",
- "\n",
- "class Momentum(Scheduler):\n",
- " def __init__(self, eta: float, momentum: float):\n",
- " super().__init__(eta)\n",
- " self.momentum = momentum\n",
- " self.change = 0\n",
- "\n",
- " def update_change(self, gradient):\n",
- " self.change = self.momentum * self.change + self.eta * gradient\n",
- " return self.change\n",
- "\n",
- " def reset(self):\n",
- " pass\n",
- "\n",
- "\n",
- "class Adagrad(Scheduler):\n",
- " def __init__(self, eta):\n",
- " super().__init__(eta)\n",
- " self.G_t = None\n",
- "\n",
- " def update_change(self, gradient):\n",
- " delta = 1e-8 # avoid division ny zero\n",
- "\n",
- " if self.G_t is None:\n",
- " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n",
- "\n",
- " self.G_t += gradient @ gradient.T\n",
- "\n",
- " G_t_inverse = 1 / (\n",
- " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n",
- " )\n",
- " return self.eta * gradient * G_t_inverse\n",
- "\n",
- " def reset(self):\n",
- " self.G_t = None\n",
- "\n",
- "\n",
- "class AdagradMomentum(Scheduler):\n",
- " def __init__(self, eta, momentum):\n",
- " super().__init__(eta)\n",
- " self.G_t = None\n",
- " self.momentum = momentum\n",
- " self.change = 0\n",
- "\n",
- " def update_change(self, gradient):\n",
- " delta = 1e-8 # avoid division ny zero\n",
- "\n",
- " if self.G_t is None:\n",
- " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n",
- "\n",
- " self.G_t += gradient @ gradient.T\n",
- "\n",
- " G_t_inverse = 1 / (\n",
- " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n",
- " )\n",
- " self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n",
- " return self.change\n",
- "\n",
- " def reset(self):\n",
- " self.G_t = None\n",
- "\n",
- "\n",
- "class RMS_prop(Scheduler):\n",
- " def __init__(self, eta, rho):\n",
- " super().__init__(eta)\n",
- " self.rho = rho\n",
- " self.second = 0.0\n",
- "\n",
- " def update_change(self, gradient):\n",
- " delta = 1e-8 # avoid division ny zero\n",
- " self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n",
- " return self.eta * gradient / (np.sqrt(self.second + delta))\n",
- "\n",
- " def reset(self):\n",
- " self.second = 0.0\n",
- "\n",
- "\n",
- "class Adam(Scheduler):\n",
- " def __init__(self, eta, rho, rho2):\n",
- " super().__init__(eta)\n",
- " self.rho = rho\n",
- " self.rho2 = rho2\n",
- " self.moment = 0\n",
- " self.second = 0\n",
- " self.n_epochs = 1\n",
- "\n",
- " def update_change(self, gradient):\n",
- " delta = 1e-8 # avoid division ny zero\n",
- "\n",
- " self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n",
- " self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n",
- "\n",
- " moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n",
- " second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n",
- "\n",
- " return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n",
- "\n",
- " def reset(self):\n",
- " self.n_epochs += 1\n",
- " self.moment = 0\n",
- " self.second = 0"
+ "from sklearn.datasets import make_classification\n",
+ "X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, n_redundant=5, n_classes=2, random_state=42)"
]
},
{
"cell_type": "markdown",
- "id": "1a9bcb3e",
- "metadata": {},
+ "id": "0068b032",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "### Usage of the above learning rate schedulers\n",
+ "You can use this option for the multiclass case as well, see the next exercise.\n",
+ "If you prefer to study other binary classification datasets, feel free\n",
+ "to replace the above suggestions with your own dataset.\n",
"\n",
- "To initalize a scheduler, simply create the object and pass in the\n",
- "necessary parameters such as the learning rate and the momentum as\n",
- "shown below. As the Scheduler class is an abstract class it should not\n",
- "called directly, and will raise an error upon usage."
+ "Make plots of the confusion matrix, the ROC curve and the cumulative gain curve."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c45f5b41",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "### Exercise c) week 43\n",
+ "\n",
+ "As a multiclass problem, we will use the Iris data set discussed in\n",
+ "the exercises from weeks 41 and 42. This is a three-class data set and\n",
+ "you can set it up using **scikit-learn**,"
]
},
{
"cell_type": "code",
"execution_count": 4,
- "id": "86013cb4",
- "metadata": {},
+ "id": "3b045d56",
+ "metadata": {
+ "collapsed": false,
+ "editable": true,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
- "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
- "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)"
+ "from sklearn.datasets import load_iris\n",
+ "iris = load_iris()\n",
+ "X = iris.data # Features\n",
+ "y = iris.target # Target labels"
]
},
{
"cell_type": "markdown",
- "id": "535331f6",
- "metadata": {},
+ "id": "14cc859c",
+ "metadata": {
+ "editable": true
+ },
"source": [
- "Here is a small example for how a segment of code using schedulers\n",
- "could look. Switching out the schedulers is simple."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "7e0f6b5a",
- "metadata": {},
- "outputs": [],
- "source": [
- "weights = np.ones((3,3))\n",
- "print(f\"Before scheduler:\\n{weights=}\")\n",
- "\n",
- "epochs = 10\n",
- "for e in range(epochs):\n",
- " gradient = np.random.rand(3, 3)\n",
- " change = adam_scheduler.update_change(gradient)\n",
- " weights = weights - change\n",
- " adam_scheduler.reset()\n",
- "\n",
- "print(f\"\\nAfter scheduler:\\n{weights=}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f018ae57",
- "metadata": {},
- "source": [
- "### Cost functions\n",
- "\n",
- "Here we discuss cost functions that can be used when creating the\n",
- "neural network. Every cost function takes the target vector as its\n",
- "parameter, and returns a function valued only at $x$ such that it may\n",
- "easily be differentiated."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "c13507bf",
- "metadata": {},
- "outputs": [],
- "source": [
- "import autograd.numpy as np\n",
- "\n",
- "def CostOLS(target):\n",
- " \n",
- " def func(X):\n",
- " return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n",
- "\n",
- " return func\n",
- "\n",
- "\n",
- "def CostLogReg(target):\n",
- "\n",
- " def func(X):\n",
- " \n",
- " return -(1.0 / target.shape[0]) * np.sum(\n",
- " (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n",
- " )\n",
- "\n",
- " return func\n",
- "\n",
- "\n",
- "def CostCrossEntropy(target):\n",
- " \n",
- " def func(X):\n",
- " return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n",
- "\n",
- " return func"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6dab17bc",
- "metadata": {},
- "source": [
- "Below we give a short example of how these cost function may be used\n",
- "to obtain results if you wish to test them out on your own using\n",
- "AutoGrad's automatics differentiation."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "a5dbba01",
- "metadata": {},
- "outputs": [],
- "source": [
- "from autograd import grad\n",
- "\n",
- "target = np.array([[1, 2, 3]]).T\n",
- "a = np.array([[4, 5, 6]]).T\n",
- "\n",
- "cost_func = CostCrossEntropy\n",
- "cost_func_derivative = grad(cost_func(target))\n",
- "\n",
- "valued_at_a = cost_func_derivative(a)\n",
- "print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b55d31d4",
- "metadata": {},
- "source": [
- "### Activation functions\n",
- "\n",
- "Finally, before we look at the neural network, we will look at the\n",
- "activation functions which can be specified between the hidden layers\n",
- "and as the output function. Each function can be valued for any given\n",
- "vector or matrix X, and can be differentiated via derivate()."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "b3e045a6",
- "metadata": {},
- "outputs": [],
- "source": [
- "import autograd.numpy as np\n",
- "from autograd import elementwise_grad\n",
- "\n",
- "def identity(X):\n",
- " return X\n",
- "\n",
- "\n",
- "def sigmoid(X):\n",
- " try:\n",
- " return 1.0 / (1 + np.exp(-X))\n",
- " except FloatingPointError:\n",
- " return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n",
- "\n",
- "\n",
- "def softmax(X):\n",
- " X = X - np.max(X, axis=-1, keepdims=True)\n",
- " delta = 10e-10\n",
- " return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n",
- "\n",
- "\n",
- "def RELU(X):\n",
- " return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n",
- "\n",
- "\n",
- "def LRELU(X):\n",
- " delta = 10e-4\n",
- " return np.where(X > np.zeros(X.shape), X, delta * X)\n",
- "\n",
- "\n",
- "def derivate(func):\n",
- " if func.__name__ == \"RELU\":\n",
- "\n",
- " def func(X):\n",
- " return np.where(X > 0, 1, 0)\n",
- "\n",
- " return func\n",
- "\n",
- " elif func.__name__ == \"LRELU\":\n",
- "\n",
- " def func(X):\n",
- " delta = 10e-4\n",
- " return np.where(X > 0, 1, delta)\n",
- "\n",
- " return func\n",
- "\n",
- " else:\n",
- " return elementwise_grad(func)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c0189342",
- "metadata": {},
- "source": [
- "Below follows a short demonstration of how to use an activation\n",
- "function. The derivative of the activation function will be important\n",
- "when calculating the output delta term during backpropagation. Note\n",
- "that derivate() can also be used for cost functions for a more\n",
- "generalized approach."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "640aa861",
- "metadata": {},
- "outputs": [],
- "source": [
- "z = np.array([[4, 5, 6]]).T\n",
- "print(f\"Input to activation function:\\n{z}\")\n",
- "\n",
- "act_func = sigmoid\n",
- "a = act_func(z)\n",
- "print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n",
- "\n",
- "act_func_derivative = derivate(act_func)\n",
- "valued_at_z = act_func_derivative(a)\n",
- "print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1007ccdd",
- "metadata": {},
- "source": [
- "### The Neural Network\n",
- "\n",
- "Now that we have gotten a good understanding of the implementation of\n",
- "some important components, we can take a look at an object oriented\n",
- "implementation of a feed forward neural network. The feed forward\n",
- "neural network has been implemented as a class named FFNN, which can\n",
- "be initiated as a regressor or classifier dependant on the choice of\n",
- "cost function. The FFNN can have any number of input nodes, hidden\n",
- "layers with any amount of hidden nodes, and any amount of output nodes\n",
- "meaning it can perform multiclass classification as well as binary\n",
- "classification and regression problems. Although there is a lot of\n",
- "code present, it makes for an easy to use and generalizeable interface\n",
- "for creating many types of neural networks as will be demonstrated\n",
- "below."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "9584a2da",
- "metadata": {},
- "outputs": [],
- "source": [
- "import math\n",
- "import autograd.numpy as np\n",
- "import sys\n",
- "import warnings\n",
- "from autograd import grad, elementwise_grad\n",
- "from random import random, seed\n",
- "from copy import deepcopy, copy\n",
- "from typing import Tuple, Callable\n",
- "from sklearn.utils import resample\n",
- "\n",
- "warnings.simplefilter(\"error\")\n",
- "\n",
- "\n",
- "class FFNN:\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Feed Forward Neural Network with interface enabling flexible design of a\n",
- " nerual networks architecture and the specification of activation function\n",
- " in the hidden layers and output layer respectively. This model can be used\n",
- " for both regression and classification problems, depending on the output function.\n",
- "\n",
- " Attributes:\n",
- " ------------\n",
- " I dimensions (tuple[int]): A list of positive integers, which specifies the\n",
- " number of nodes in each of the networks layers. The first integer in the array\n",
- " defines the number of nodes in the input layer, the second integer defines number\n",
- " of nodes in the first hidden layer and so on until the last number, which\n",
- " specifies the number of nodes in the output layer.\n",
- " II hidden_func (Callable): The activation function for the hidden layers\n",
- " III output_func (Callable): The activation function for the output layer\n",
- " IV cost_func (Callable): Our cost function\n",
- " V seed (int): Sets random seed, makes results reproducible\n",
- " \"\"\"\n",
- "\n",
- " def __init__(\n",
- " self,\n",
- " dimensions: tuple[int],\n",
- " hidden_func: Callable = sigmoid,\n",
- " output_func: Callable = lambda x: x,\n",
- " cost_func: Callable = CostOLS,\n",
- " seed: int = None,\n",
- " ):\n",
- " self.dimensions = dimensions\n",
- " self.hidden_func = hidden_func\n",
- " self.output_func = output_func\n",
- " self.cost_func = cost_func\n",
- " self.seed = seed\n",
- " self.weights = list()\n",
- " self.schedulers_weight = list()\n",
- " self.schedulers_bias = list()\n",
- " self.a_matrices = list()\n",
- " self.z_matrices = list()\n",
- " self.classification = None\n",
- "\n",
- " self.reset_weights()\n",
- " self._set_classification()\n",
- "\n",
- " def fit(\n",
- " self,\n",
- " X: np.ndarray,\n",
- " t: np.ndarray,\n",
- " scheduler: Scheduler,\n",
- " batches: int = 1,\n",
- " epochs: int = 100,\n",
- " lam: float = 0,\n",
- " X_val: np.ndarray = None,\n",
- " t_val: np.ndarray = None,\n",
- " ):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " This function performs the training the neural network by performing the feedforward and backpropagation\n",
- " algorithm to update the networks weights.\n",
- "\n",
- " Parameters:\n",
- " ------------\n",
- " I X (np.ndarray) : training data\n",
- " II t (np.ndarray) : target data\n",
- " III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n",
- " IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n",
- "\n",
- " Optional Parameters:\n",
- " ------------\n",
- " V batches (int) : number of batches the datasets are split into, default equal to 1\n",
- " VI epochs (int) : number of iterations used to train the network, default equal to 100\n",
- " VII lam (float) : regularization hyperparameter lambda\n",
- " VIII X_val (np.ndarray) : validation set\n",
- " IX t_val (np.ndarray) : validation target set\n",
- "\n",
- " Returns:\n",
- " ------------\n",
- " I scores (dict) : A dictionary containing the performance metrics of the model.\n",
- " The number of the metrics depends on the parameters passed to the fit-function.\n",
- "\n",
- " \"\"\"\n",
- "\n",
- " # setup \n",
- " if self.seed is not None:\n",
- " np.random.seed(self.seed)\n",
- "\n",
- " val_set = False\n",
- " if X_val is not None and t_val is not None:\n",
- " val_set = True\n",
- "\n",
- " # creating arrays for score metrics\n",
- " train_errors = np.empty(epochs)\n",
- " train_errors.fill(np.nan)\n",
- " val_errors = np.empty(epochs)\n",
- " val_errors.fill(np.nan)\n",
- "\n",
- " train_accs = np.empty(epochs)\n",
- " train_accs.fill(np.nan)\n",
- " val_accs = np.empty(epochs)\n",
- " val_accs.fill(np.nan)\n",
- "\n",
- " self.schedulers_weight = list()\n",
- " self.schedulers_bias = list()\n",
- "\n",
- " batch_size = X.shape[0] // batches\n",
- "\n",
- " X, t = resample(X, t)\n",
- "\n",
- " # this function returns a function valued only at X\n",
- " cost_function_train = self.cost_func(t)\n",
- " if val_set:\n",
- " cost_function_val = self.cost_func(t_val)\n",
- "\n",
- " # create schedulers for each weight matrix\n",
- " for i in range(len(self.weights)):\n",
- " self.schedulers_weight.append(copy(scheduler))\n",
- " self.schedulers_bias.append(copy(scheduler))\n",
- "\n",
- " print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n",
- "\n",
- " try:\n",
- " for e in range(epochs):\n",
- " for i in range(batches):\n",
- " # allows for minibatch gradient descent\n",
- " if i == batches - 1:\n",
- " # If the for loop has reached the last batch, take all thats left\n",
- " X_batch = X[i * batch_size :, :]\n",
- " t_batch = t[i * batch_size :, :]\n",
- " else:\n",
- " X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n",
- " t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n",
- "\n",
- " self._feedforward(X_batch)\n",
- " self._backpropagate(X_batch, t_batch, lam)\n",
- "\n",
- " # reset schedulers for each epoch (some schedulers pass in this call)\n",
- " for scheduler in self.schedulers_weight:\n",
- " scheduler.reset()\n",
- "\n",
- " for scheduler in self.schedulers_bias:\n",
- " scheduler.reset()\n",
- "\n",
- " # computing performance metrics\n",
- " pred_train = self.predict(X)\n",
- " train_error = cost_function_train(pred_train)\n",
- "\n",
- " train_errors[e] = train_error\n",
- " if val_set:\n",
- " \n",
- " pred_val = self.predict(X_val)\n",
- " val_error = cost_function_val(pred_val)\n",
- " val_errors[e] = val_error\n",
- "\n",
- " if self.classification:\n",
- " train_acc = self._accuracy(self.predict(X), t)\n",
- " train_accs[e] = train_acc\n",
- " if val_set:\n",
- " val_acc = self._accuracy(pred_val, t_val)\n",
- " val_accs[e] = val_acc\n",
- "\n",
- " # printing progress bar\n",
- " progression = e / epochs\n",
- " print_length = self._progress_bar(\n",
- " progression,\n",
- " train_error=train_errors[e],\n",
- " train_acc=train_accs[e],\n",
- " val_error=val_errors[e],\n",
- " val_acc=val_accs[e],\n",
- " )\n",
- " except KeyboardInterrupt:\n",
- " # allows for stopping training at any point and seeing the result\n",
- " pass\n",
- "\n",
- " # visualization of training progression (similiar to tensorflow progression bar)\n",
- " sys.stdout.write(\"\\r\" + \" \" * print_length)\n",
- " sys.stdout.flush()\n",
- " self._progress_bar(\n",
- " 1,\n",
- " train_error=train_errors[e],\n",
- " train_acc=train_accs[e],\n",
- " val_error=val_errors[e],\n",
- " val_acc=val_accs[e],\n",
- " )\n",
- " sys.stdout.write(\"\")\n",
- "\n",
- " # return performance metrics for the entire run\n",
- " scores = dict()\n",
- "\n",
- " scores[\"train_errors\"] = train_errors\n",
- "\n",
- " if val_set:\n",
- " scores[\"val_errors\"] = val_errors\n",
- "\n",
- " if self.classification:\n",
- " scores[\"train_accs\"] = train_accs\n",
- "\n",
- " if val_set:\n",
- " scores[\"val_accs\"] = val_accs\n",
- "\n",
- " return scores\n",
- "\n",
- " def predict(self, X: np.ndarray, *, threshold=0.5):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Performs prediction after training of the network has been finished.\n",
- "\n",
- " Parameters:\n",
- " ------------\n",
- " I X (np.ndarray): The design matrix, with n rows of p features each\n",
- "\n",
- " Optional Parameters:\n",
- " ------------\n",
- " II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n",
- " in classification problems\n",
- "\n",
- " Returns:\n",
- " ------------\n",
- " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n",
- " This vector is thresholded if regression=False, meaning that classification results\n",
- " in a vector of 1s and 0s, while regressions in an array of decimal numbers\n",
- "\n",
- " \"\"\"\n",
- "\n",
- " predict = self._feedforward(X)\n",
- "\n",
- " if self.classification:\n",
- " return np.where(predict > threshold, 1, 0)\n",
- " else:\n",
- " return predict\n",
- "\n",
- " def reset_weights(self):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Resets/Reinitializes the weights in order to train the network for a new problem.\n",
- "\n",
- " \"\"\"\n",
- " if self.seed is not None:\n",
- " np.random.seed(self.seed)\n",
- "\n",
- " self.weights = list()\n",
- " for i in range(len(self.dimensions) - 1):\n",
- " weight_array = np.random.randn(\n",
- " self.dimensions[i] + 1, self.dimensions[i + 1]\n",
- " )\n",
- " weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n",
- "\n",
- " self.weights.append(weight_array)\n",
- "\n",
- " def _feedforward(self, X: np.ndarray):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Calculates the activation of each layer starting at the input and ending at the output.\n",
- " Each following activation is calculated from a weighted sum of each of the preceeding\n",
- " activations (except in the case of the input layer).\n",
- "\n",
- " Parameters:\n",
- " ------------\n",
- " I X (np.ndarray): The design matrix, with n rows of p features each\n",
- "\n",
- " Returns:\n",
- " ------------\n",
- " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n",
- " \"\"\"\n",
- "\n",
- " # reset matrices\n",
- " self.a_matrices = list()\n",
- " self.z_matrices = list()\n",
- "\n",
- " # if X is just a vector, make it into a matrix\n",
- " if len(X.shape) == 1:\n",
- " X = X.reshape((1, X.shape[0]))\n",
- "\n",
- " # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n",
- " # to add bias to our data\n",
- " bias = np.ones((X.shape[0], 1)) * 0.01\n",
- " X = np.hstack([bias, X])\n",
- "\n",
- " # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n",
- " # exponent indicates layer number).\n",
- " a = X\n",
- " self.a_matrices.append(a)\n",
- " self.z_matrices.append(a)\n",
- "\n",
- " # The feed forward algorithm\n",
- " for i in range(len(self.weights)):\n",
- " if i < len(self.weights) - 1:\n",
- " z = a @ self.weights[i]\n",
- " self.z_matrices.append(z)\n",
- " a = self.hidden_func(z)\n",
- " # bias column again added to the data here\n",
- " bias = np.ones((a.shape[0], 1)) * 0.01\n",
- " a = np.hstack([bias, a])\n",
- " self.a_matrices.append(a)\n",
- " else:\n",
- " try:\n",
- " # a^L, the nodes in our output layers\n",
- " z = a @ self.weights[i]\n",
- " a = self.output_func(z)\n",
- " self.a_matrices.append(a)\n",
- " self.z_matrices.append(z)\n",
- " except Exception as OverflowError:\n",
- " print(\n",
- " \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n",
- " )\n",
- "\n",
- " # this will be a^L\n",
- " return a\n",
- "\n",
- " def _backpropagate(self, X, t, lam):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Performs the backpropagation algorithm. In other words, this method\n",
- " calculates the gradient of all the layers starting at the\n",
- " output layer, and moving from right to left accumulates the gradient until\n",
- " the input layer is reached. Each layers respective weights are updated while\n",
- " the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n",
- "\n",
- " Parameters:\n",
- " ------------\n",
- " I X (np.ndarray): The design matrix, with n rows of p features each.\n",
- " II t (np.ndarray): The target vector, with n rows of p targets.\n",
- " III lam (float32): regularization parameter used to punish the weights in case of overfitting\n",
- "\n",
- " Returns:\n",
- " ------------\n",
- " No return value.\n",
- "\n",
- " \"\"\"\n",
- " out_derivative = derivate(self.output_func)\n",
- " hidden_derivative = derivate(self.hidden_func)\n",
- "\n",
- " for i in range(len(self.weights) - 1, -1, -1):\n",
- " # delta terms for output\n",
- " if i == len(self.weights) - 1:\n",
- " # for multi-class classification\n",
- " if (\n",
- " self.output_func.__name__ == \"softmax\"\n",
- " ):\n",
- " delta_matrix = self.a_matrices[i + 1] - t\n",
- " # for single class classification\n",
- " else:\n",
- " cost_func_derivative = grad(self.cost_func(t))\n",
- " delta_matrix = out_derivative(\n",
- " self.z_matrices[i + 1]\n",
- " ) * cost_func_derivative(self.a_matrices[i + 1])\n",
- "\n",
- " # delta terms for hidden layer\n",
- " else:\n",
- " delta_matrix = (\n",
- " self.weights[i + 1][1:, :] @ delta_matrix.T\n",
- " ).T * hidden_derivative(self.z_matrices[i + 1])\n",
- "\n",
- " # calculate gradient\n",
- " gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n",
- " gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n",
- " 1, delta_matrix.shape[1]\n",
- " )\n",
- "\n",
- " # regularization term\n",
- " gradient_weights += self.weights[i][1:, :] * lam\n",
- "\n",
- " # use scheduler\n",
- " update_matrix = np.vstack(\n",
- " [\n",
- " self.schedulers_bias[i].update_change(gradient_bias),\n",
- " self.schedulers_weight[i].update_change(gradient_weights),\n",
- " ]\n",
- " )\n",
- "\n",
- " # update weights and bias\n",
- " self.weights[i] -= update_matrix\n",
- "\n",
- " def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Calculates accuracy of given prediction to target\n",
- "\n",
- " Parameters:\n",
- " ------------\n",
- " I prediction (np.ndarray): vector of predicitons output network\n",
- " (1s and 0s in case of classification, and real numbers in case of regression)\n",
- " II target (np.ndarray): vector of true values (What the network ideally should predict)\n",
- "\n",
- " Returns:\n",
- " ------------\n",
- " A floating point number representing the percentage of correctly classified instances.\n",
- " \"\"\"\n",
- " assert prediction.size == target.size\n",
- " return np.average((target == prediction))\n",
- " def _set_classification(self):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Decides if FFNN acts as classifier (True) og regressor (False),\n",
- " sets self.classification during init()\n",
- " \"\"\"\n",
- " self.classification = False\n",
- " if (\n",
- " self.cost_func.__name__ == \"CostLogReg\"\n",
- " or self.cost_func.__name__ == \"CostCrossEntropy\"\n",
- " ):\n",
- " self.classification = True\n",
- "\n",
- " def _progress_bar(self, progression, **kwargs):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Displays progress of training\n",
- " \"\"\"\n",
- " print_length = 40\n",
- " num_equals = int(progression * print_length)\n",
- " num_not = print_length - num_equals\n",
- " arrow = \">\" if num_equals > 0 else \"\"\n",
- " bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n",
- " perc_print = self._format(progression * 100, decimals=5)\n",
- " line = f\" {bar} {perc_print}% \"\n",
- "\n",
- " for key in kwargs:\n",
- " if not np.isnan(kwargs[key]):\n",
- " value = self._format(kwargs[key], decimals=4)\n",
- " line += f\"| {key}: {value} \"\n",
- " sys.stdout.write(\"\\r\" + line)\n",
- " sys.stdout.flush()\n",
- " return len(line)\n",
- "\n",
- " def _format(self, value, decimals=4):\n",
- " \"\"\"\n",
- " Description:\n",
- " ------------\n",
- " Formats decimal numbers for progress bar\n",
- " \"\"\"\n",
- " if value > 0:\n",
- " v = value\n",
- " elif value < 0:\n",
- " v = -10 * value\n",
- " else:\n",
- " v = 1\n",
- " n = 1 + math.floor(math.log10(v))\n",
- " if n >= decimals - 1:\n",
- " return str(round(value))\n",
- " return f\"{value:.{decimals-n-1}f}\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9ccd1fc1",
- "metadata": {},
- "source": [
- "Before we make a model, we will quickly generate a dataset we can use\n",
- "for our linear regression problem as shown below"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "id": "7f3a5b31",
- "metadata": {},
- "outputs": [],
- "source": [
- "import autograd.numpy as np\n",
- "from sklearn.model_selection import train_test_split\n",
- "\n",
- "def SkrankeFunction(x, y):\n",
- " return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n",
- "\n",
- "def create_X(x, y, n):\n",
- " if len(x.shape) > 1:\n",
- " x = np.ravel(x)\n",
- " y = np.ravel(y)\n",
- "\n",
- " N = len(x)\n",
- " l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n",
- " X = np.ones((N, l))\n",
- "\n",
- " for i in range(1, n + 1):\n",
- " q = int((i) * (i + 1) / 2)\n",
- " for k in range(i + 1):\n",
- " X[:, q + k] = (x ** (i - k)) * (y**k)\n",
- "\n",
- " return X\n",
- "\n",
- "step=0.5\n",
- "x = np.arange(0, 1, step)\n",
- "y = np.arange(0, 1, step)\n",
- "x, y = np.meshgrid(x, y)\n",
- "target = SkrankeFunction(x, y)\n",
- "target = target.reshape(target.shape[0], 1)\n",
- "\n",
- "poly_degree=3\n",
- "X = create_X(x, y, poly_degree)\n",
- "\n",
- "X_train, X_test, t_train, t_test = train_test_split(X, target)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1ac05bb6",
- "metadata": {},
- "source": [
- "Now that we have our dataset ready for the regression, we can create\n",
- "our regressor. Note that with the seed parameter, we can make sure our\n",
- "results stay the same every time we run the neural network. For\n",
- "inititialization, we simply specify the dimensions (we wish the amount\n",
- "of input nodes to be equal to the datapoints, and the output to\n",
- "predict one value)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "id": "0f857604",
- "metadata": {},
- "outputs": [],
- "source": [
- "input_nodes = X_train.shape[1]\n",
- "output_nodes = 1\n",
- "\n",
- "linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "eeff4315",
- "metadata": {},
- "source": [
- "We then fit our model with our training data using the scheduler of our choice."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "id": "46246810",
- "metadata": {},
- "outputs": [],
- "source": [
- "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
- "\n",
- "scheduler = Constant(eta=1e-3)\n",
- "scores = linear_regression.fit(X_train, t_train, scheduler)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8c6f9954",
- "metadata": {},
- "source": [
- "Due to the progress bar we can see the MSE (train_error) throughout\n",
- "the FFNN's training. Note that the fit() function has some optional\n",
- "parameters with defualt arguments. For example, the regularization\n",
- "hyperparameter can be left ignored if not needed, and equally the FFNN\n",
- "will by default run for 100 epochs. These can easily be changed, such\n",
- "as for example:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "id": "2661939c",
- "metadata": {},
- "outputs": [],
- "source": [
- "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
- "\n",
- "scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "74c5624c",
- "metadata": {},
- "source": [
- "We see that given more epochs to train on, the regressor reaches a lower MSE.\n",
- "\n",
- "Let us then switch to a binary classification. We use a binary\n",
- "classification dataset, and follow a similar setup to the regression\n",
- "case."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "4b8eb115",
- "metadata": {},
- "outputs": [],
- "source": [
- "from sklearn.datasets import load_breast_cancer\n",
- "from sklearn.preprocessing import MinMaxScaler\n",
- "\n",
- "wisconsin = load_breast_cancer()\n",
- "X = wisconsin.data\n",
- "target = wisconsin.target\n",
- "target = target.reshape(target.shape[0], 1)\n",
- "\n",
- "X_train, X_val, t_train, t_val = train_test_split(X, target)\n",
- "\n",
- "scaler = MinMaxScaler()\n",
- "scaler.fit(X_train)\n",
- "X_train = scaler.transform(X_train)\n",
- "X_val = scaler.transform(X_val)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "2c0f92bd",
- "metadata": {},
- "outputs": [],
- "source": [
- "input_nodes = X_train.shape[1]\n",
- "output_nodes = 1\n",
- "\n",
- "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "49201ae4",
- "metadata": {},
- "source": [
- "We will now make use of our validation data by passing it into our fit function as a keyword argument"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "id": "55b5e426",
- "metadata": {},
- "outputs": [],
- "source": [
- "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
- "\n",
- "scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n",
- "scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b51762fb",
- "metadata": {},
- "source": [
- "Finally, we will create a neural network with 2 hidden layers with activation functions."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "id": "6b59e27d",
- "metadata": {},
- "outputs": [],
- "source": [
- "input_nodes = X_train.shape[1]\n",
- "hidden_nodes1 = 100\n",
- "hidden_nodes2 = 30\n",
- "output_nodes = 1\n",
- "\n",
- "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n",
- "\n",
- "neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "id": "72c87921",
- "metadata": {},
- "outputs": [],
- "source": [
- "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
- "\n",
- "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n",
- "scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ed40d7d2",
- "metadata": {},
- "source": [
- "### Multiclass classification\n",
- "\n",
- "Finally, we will demonstrate the use case of multiclass classification\n",
- "using our FFNN with the famous MNIST dataset, which contain images of\n",
- "digits between the range of 0 to 9."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "id": "315ef3fe",
- "metadata": {},
- "outputs": [],
- "source": [
- "from sklearn.datasets import load_digits\n",
- "\n",
- "def onehot(target: np.ndarray):\n",
- " onehot = np.zeros((target.size, target.max() + 1))\n",
- " onehot[np.arange(target.size), target] = 1\n",
- " return onehot\n",
- "\n",
- "digits = load_digits()\n",
- "\n",
- "X = digits.data\n",
- "target = digits.target\n",
- "target = onehot(target)\n",
- "\n",
- "input_nodes = 64\n",
- "hidden_nodes1 = 100\n",
- "hidden_nodes2 = 30\n",
- "output_nodes = 10\n",
- "\n",
- "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n",
- "\n",
- "multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)\n",
- "\n",
- "multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
- "\n",
- "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n",
- "scores = multiclass.fit(X, target, scheduler, epochs=1000)"
+ "Make plots of the confusion matrix, the ROC curve and the cumulative\n",
+ "gain curve for this (or other) multiclass data set."
]
}
],
@@ -1507,7 +639,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.10"
+ "version": "3.9.15"
}
},
"nbformat": 4,
diff --git a/doc/LectureNotes/exercisesweek43.ipynb b/doc/LectureNotes/exercisesweek43.ipynb
index 737d0d60c..f80e8787a 100644
--- a/doc/LectureNotes/exercisesweek43.ipynb
+++ b/doc/LectureNotes/exercisesweek43.ipynb
@@ -32,15 +32,15 @@
"editable": true
},
"source": [
- "# Overarching aims of the exercises weeks 43 and 44\n",
+ "# Overarching aims of the exercises for week 43\n",
"\n",
"The aim of the exercises this week is to gain some confidence with\n",
"ways to visualize the results of a classification problem. We will\n",
"target three ways of setting up the analysis. The first and simplest\n",
"one is the\n",
- "1. so-called confusion matrix, and the next is the\n",
+ "1. so-called confusion matrix. The next one is the so-called\n",
"\n",
- "2. ROC curve and finally the\n",
+ "2. ROC curve. Finally we have the\n",
"\n",
"3. Cumulative gain curve.\n",
"\n",
@@ -446,7 +446,10 @@
"id": "be9ff0b9",
"metadata": {
"collapsed": false,
- "editable": true
+ "editable": true,
+ "jupyter": {
+ "outputs_hidden": false
+ }
},
"outputs": [],
"source": [
@@ -518,7 +521,10 @@
"id": "d20bb8be",
"metadata": {
"collapsed": false,
- "editable": true
+ "editable": true,
+ "jupyter": {
+ "outputs_hidden": false
+ }
},
"outputs": [],
"source": [
@@ -547,7 +553,10 @@
"id": "d271f0ba",
"metadata": {
"collapsed": false,
- "editable": true
+ "editable": true,
+ "jupyter": {
+ "outputs_hidden": false
+ }
},
"outputs": [],
"source": [
@@ -589,7 +598,10 @@
"id": "3b045d56",
"metadata": {
"collapsed": false,
- "editable": true
+ "editable": true,
+ "jupyter": {
+ "outputs_hidden": false
+ }
},
"outputs": [],
"source": [
@@ -611,7 +623,25 @@
]
}
],
- "metadata": {},
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.15"
+ }
+ },
"nbformat": 4,
"nbformat_minor": 5
}