update book
This commit is contained in:
Morten Hjorth-Jensen
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
File diff suppressed because one or more lines are too long
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ddffe3be",
|
||||
"id": "31a10e2b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cde0bbff",
|
||||
"id": "019daa83",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f48c6388",
|
||||
"id": "6ecf1c2b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -67,7 +67,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ba6964ff",
|
||||
"id": "d80a4db8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -77,7 +77,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4d810bbf",
|
||||
"id": "1ded5c92",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -96,7 +96,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "39f540b5",
|
||||
"id": "6dea0513",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -153,7 +153,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "09d24c34",
|
||||
"id": "440bf579",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -165,7 +165,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7c84429f",
|
||||
"id": "683b1da2",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -180,7 +180,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "ac8fda39",
|
||||
"id": "38be79e0",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -233,7 +233,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b9da31b6",
|
||||
"id": "081fbe5c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -248,7 +248,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "afe2e276",
|
||||
"id": "a9798c07",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -268,7 +268,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "5707b319",
|
||||
"id": "0a7e4e2e",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -322,7 +322,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f067ea7",
|
||||
"id": "dcd07cb8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -337,7 +337,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "cfe2ed78",
|
||||
"id": "65061d95",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -364,7 +364,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "97a315dc",
|
||||
"id": "50492453",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -378,7 +378,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "9363662a",
|
||||
"id": "08a2ab17",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -423,7 +423,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "298800d9",
|
||||
"id": "5408b7dd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -448,7 +448,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "9db77f36",
|
||||
"id": "76c3d259",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -460,7 +460,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2872089d",
|
||||
"id": "83903231",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -471,7 +471,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "926cdfd7",
|
||||
"id": "58ccd6c3",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -483,7 +483,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6bd13984",
|
||||
"id": "adda54cb",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -496,7 +496,327 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "55242d6d",
|
||||
"id": "8799c034",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Other ways of presenting a classification problem\n",
|
||||
"\n",
|
||||
"For a binary classifcation matrix, the so-called **confusion matrix**, is often used. It can also be extended to more catgeories/classes as well.\n",
|
||||
"The following quantities are then used\n",
|
||||
"1. positive condition number $P$, which represents the number of real positive cases in the data (output one/true etc)\n",
|
||||
"\n",
|
||||
"2. The condition negative number $N$ which is the number of negative cases (ouput zero/false etc)\n",
|
||||
"\n",
|
||||
"3. The true positive number $TP$ which represents whether a positive test result has been correctly classified (the application of our trained model on a test data set)\n",
|
||||
"\n",
|
||||
"4. The true negative $TN$ number which represents whether a negative test has been correctly classified\n",
|
||||
"\n",
|
||||
"5. The false positive $FP$ number, a so-called type I error which tells us about the fraction of positive test result which are wrongly classified\n",
|
||||
"\n",
|
||||
"6. A false negative $FN$ number, a so-called type II error which, should be pretty obvious, indicates if a negative test has been wrongly classified.\n",
|
||||
"\n",
|
||||
"It is is easy to think in terms of illness. You could think of the above as\n",
|
||||
"1. True positive: Sick people correctly identified as sick\n",
|
||||
"\n",
|
||||
"2. False positive: Healthy people incorrectly identified as sick\n",
|
||||
"\n",
|
||||
"3. True negative: Healthy people correctly identified as healthy\n",
|
||||
"\n",
|
||||
"4. False negative: Sick people incorrectly identified as healthy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dfbe0571",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Combinations of classification results\n",
|
||||
"\n",
|
||||
"It is common in the literature to define various combinations the above numbers. The most commonly used are\n",
|
||||
"\n",
|
||||
"**Sensitivity, recall, hit rate, or true positive rate $TPR$. It is the probability of a positive test result, conditioned on the individual truly being positive.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "65f7c63d",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {TPR} ={\\frac {\\mathrm {TP} }{\\mathrm {P} }}={\\frac {\\mathrm {TP} }{\\mathrm {TP} +\\mathrm {FN} }}=1-\\mathrm {FNR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2e674d10",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"The $TPR$ defines how many correct positive results occur among all positive samples available during the test\n",
|
||||
"\n",
|
||||
"**Miss rate or false negative rate $FNR$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "65577837",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {FNR} ={\\frac {\\mathrm {FN} }{\\mathrm {P} }}={\\frac {\\mathrm {FN} }{\\mathrm {FN} +\\mathrm {TP} }} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e5e8802d",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**Specificity, selectivity or true negative rate $TNR$. It is the probability of a negative test result, conditioned on the individual truly being negative.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "07c61591",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {TNR} ={\\frac {\\mathrm {TN} }{\\mathrm {N} }}={\\frac {\\mathrm {TN} }{\\mathrm {TN} +\\mathrm {FP} }}=1-\\mathrm {FPR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b3f61bb3",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"with the fall-out false positive rate"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6544105e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {FPR} ={\\frac {\\mathrm {FP} }{\\mathrm {N} }}={\\frac {\\mathrm {FP} }{\\mathrm {FP} +\\mathrm {TN} }}=1-\\mathrm {TNR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0b571e85",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"The $FPR$ defines how many incorrect positive results occur among\n",
|
||||
"all negative samples available during the test."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "17d6872a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Positive and negative prediction values\n",
|
||||
"\n",
|
||||
"The positive and negative predictive values \n",
|
||||
"are the proportions of positive and negative results in statistics and\n",
|
||||
"diagnostic tests that are true positive and true negative results,\n",
|
||||
"respectively.[1] The PPV and NPV describe the performance of a\n",
|
||||
"diagnostic test or other statistical measure. A high result can be\n",
|
||||
"interpreted as indicating the accuracy of such a statistic.\n",
|
||||
"\n",
|
||||
"**Precision or positive predictive value $PPV$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "122f7e6c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {PPV} ={\\frac {\\mathrm {TP} }{\\mathrm {TP} +\\mathrm {FP} }}=1-\\mathrm {FDR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f44a5e78",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**Negative predictive value $NPV$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "86fa9670",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {NPV} ={\\frac {\\mathrm {TN} }{\\mathrm {TN} +\\mathrm {FN} }}=1-\\mathrm {FOR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a30e358d",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Other quantities\n",
|
||||
"\n",
|
||||
"**False discovery rate $FDR$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3f0de6de",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {FDR} ={\\frac {\\mathrm {FP} }{\\mathrm {FP} +\\mathrm {TP} }}=1-\\mathrm {PPV} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0b4cc93f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**False omission rate $FOR$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4866ec0a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {FOR} ={\\frac {\\mathrm {FN} }{\\mathrm {FN} +\\mathrm {TN} }}=1-\\mathrm {NPV} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9b1c8c45",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## $F_1$ score\n",
|
||||
"\n",
|
||||
"In statistical analysis of binary classification, the F-score or\n",
|
||||
"F-measure is a measure of a test's accuracy. It is calculated from the\n",
|
||||
"precision and recall of the test, where the precision is the number of\n",
|
||||
"true positive results divided by the number of all positive results,\n",
|
||||
"including those not identified correctly, and the recall is the number\n",
|
||||
"of true positive results divided by the number of all samples that\n",
|
||||
"should have been identified as positive. Precision is also known as\n",
|
||||
"positive predictive value, and recall is also known as sensitivity in\n",
|
||||
"diagnostic binary classification.\n",
|
||||
"\n",
|
||||
"The F1 score is the harmonic mean of the precision and recall. It thus\n",
|
||||
"symmetrically represents both precision and recall in one metric. The\n",
|
||||
"highest possible value of an F-score is 1.0, indicating perfect\n",
|
||||
"precision and recall, and the lowest possible value is 0, if either\n",
|
||||
"precision or recall are zero.\n",
|
||||
"\n",
|
||||
"It is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0e651a1f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {F} _{1}=2\\times {\\frac {\\mathrm {PPV} \\times \\mathrm {TPR} }{\\mathrm {PPV} +\\mathrm {TPR} }}={\\frac {2\\mathrm {TP} }{2\\mathrm {TP} +\\mathrm {FP} +\\mathrm {FN} }}}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d83efe41",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## ROC curve\n",
|
||||
"\n",
|
||||
"A receiver operating characteristic curve, or ROC curve, is a\n",
|
||||
"graphical plot that illustrates the performance of a binary classifier\n",
|
||||
"model at varying threshold values.\n",
|
||||
"\n",
|
||||
"The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.\n",
|
||||
"\n",
|
||||
"To draw a ROC curve, only the true positive rate (TPR) and false\n",
|
||||
"positive rate (FPR) are needed (as functions of some classifier\n",
|
||||
"parameter). The TPR defines how many correct positive results occur\n",
|
||||
"among all positive samples available during the test. FPR, on the\n",
|
||||
"other hand, defines how many incorrect positive results occur among\n",
|
||||
"all negative samples available during the test.\n",
|
||||
"\n",
|
||||
"See <https://en.wikipedia.org/wiki/Receiver_operating_characteristic> for more discussions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d088215b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Cumulative gain curve\n",
|
||||
"\n",
|
||||
"The cumulative gain curve is a performance evaluation used typically for binary classification problems.\n",
|
||||
"It plots the $TPR$ True Positive Rate or Sensitivity (which represents the \n",
|
||||
"fraction of examples correctly classified\n",
|
||||
"against Predictive Positive Rate, which represents \n",
|
||||
"the fraction of positively predicted examples.\n",
|
||||
"\n",
|
||||
"The examples below show the confusion matrix, the ROC curve and the cumulative gain for the Wisconsin cancer data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5c0bcd1f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -507,7 +827,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "00c15782",
|
||||
"id": "2a6f80df",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -550,40 +870,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7a8eeb7",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## ROC curve\n",
|
||||
"\n",
|
||||
"A receiver operating characteristic curve, or ROC curve, is a\n",
|
||||
"graphical plot that illustrates the performance of a binary classifier\n",
|
||||
"model at varying threshold values.\n",
|
||||
"\n",
|
||||
"The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.\n",
|
||||
"See <https://en.wikipedia.org/wiki/Receiver_operating_characteristic> for more discussions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6414e652",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Cumulative gain curve\n",
|
||||
"\n",
|
||||
"The cumulative gain curve is a performance evaluation used typically for binary classification problems.\n",
|
||||
"It plots the $TPR$ True Positive Rate or Sensitivity (which represents the \n",
|
||||
"fraction of examples correctly classified\n",
|
||||
"against Predictive Positive Rate, which represents \n",
|
||||
"the fraction of positively predicted examples."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7646c025",
|
||||
"id": "58e5b521",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -593,7 +880,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f7b4b23b",
|
||||
"id": "4ebe5a91",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -620,7 +907,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "741bf972",
|
||||
"id": "182f425e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -631,7 +918,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "f9060416",
|
||||
"id": "8b3ca785",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -704,7 +991,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e0342b0b",
|
||||
"id": "b0c12b4c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -724,7 +1011,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "58fdc710",
|
||||
"id": "d81020f6",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -740,7 +1027,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f793c129",
|
||||
"id": "2d7453c9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -762,7 +1049,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7446349",
|
||||
"id": "fc4a082e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -780,7 +1067,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fac85e06",
|
||||
"id": "c1106ad9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -826,7 +1113,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7e3fbdb7",
|
||||
"id": "b139ef7b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -847,7 +1134,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f29008fb",
|
||||
"id": "70c95078",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -908,7 +1195,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dedfb5dd",
|
||||
"id": "2ca24031",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -933,7 +1220,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2abfa20e",
|
||||
"id": "93648979",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -952,7 +1239,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "abc4dabb",
|
||||
"id": "b8806dcd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -976,7 +1263,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "67e1ca86",
|
||||
"id": "b0edad40",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1058,7 +1345,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bc04ebe9",
|
||||
"id": "58fd18f9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1074,7 +1361,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "a5882cbb",
|
||||
"id": "a604caa5",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1113,7 +1400,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "18221ec4",
|
||||
"id": "545336c9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1157,7 +1444,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "d4982f3a",
|
||||
"id": "3ef3ed13",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1240,7 +1527,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a146c2c5",
|
||||
"id": "8a8b29b8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1251,7 +1538,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "2f7691d6",
|
||||
"id": "d840beea",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1353,7 +1640,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "618a775c",
|
||||
"id": "2e5b339f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1375,7 +1662,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "5f955304",
|
||||
"id": "370e799d",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1472,7 +1759,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ae8bb1a0",
|
||||
"id": "d8f62fc4",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1498,7 +1785,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "701451d4",
|
||||
"id": "4ea36ea9",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
|
||||
@@ -1035,74 +1035,6 @@ Accuracy score on data set: 1.0
|
||||
Learning rate = 0.1
|
||||
Lambda = 0.01
|
||||
Accuracy score on data set: 1.0
|
||||
|
||||
Learning rate = 0.1
|
||||
Lambda = 0.1
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 0.1
|
||||
Lambda = 1.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 0.1
|
||||
Lambda = 10.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 1e-05
|
||||
Accuracy score on data set: 0.75
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.0001
|
||||
Accuracy score on data set: 0.75
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.001
|
||||
Accuracy score on data set: 0.75
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.01
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.1
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 1.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 10.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 1e-05
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.0001
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.001
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.01
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.1
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 1.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 10.0
|
||||
Accuracy score on data set: 0.5
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
@@ -1127,7 +1059,76 @@ Accuracy score on data set: 0.5
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/exercisesweek43_26_2.png" src="_images/exercisesweek43_26_2.png" />
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
|
||||
Lambda = 0.1
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 0.1
|
||||
Lambda = 1.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 0.1
|
||||
Lambda = 10.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 1e-05
|
||||
Accuracy score on data set: 0.75
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.0001
|
||||
Accuracy score on data set: 0.75
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.001
|
||||
Accuracy score on data set: 0.75
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.01
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.1
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 1.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 10.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 1e-05
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.0001
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.001
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.01
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.1
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 1.0
|
||||
Accuracy score on data set: 0.5
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 10.0
|
||||
Accuracy score on data set: 0.5
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/exercisesweek43_26_3.png" src="_images/exercisesweek43_26_3.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -5432,9 +5433,8 @@ case.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Adam: Eta=0.001, Lambda=0
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> [----------------------------------------] 0.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392
|
||||
|
||||
[----------------------------------------] 0.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> [----------------------------------------] 0.1000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -485,8 +485,31 @@ const thebe_selector_output = ".output, .cell_output"
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#other-measures-in-classification-studies-cancer-data-again">
|
||||
Other measures in classification studies: Cancer Data again
|
||||
<a class="reference internal nav-link" href="#other-ways-of-presenting-a-classification-problem">
|
||||
Other ways of presenting a classification problem
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#combinations-of-classification-results">
|
||||
Combinations of classification results
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#positive-and-negative-prediction-values">
|
||||
Positive and negative prediction values
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#other-quantities">
|
||||
Other quantities
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#f-1-score">
|
||||
<span class="math notranslate nohighlight">
|
||||
\(F_1\)
|
||||
</span>
|
||||
score
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
@@ -499,6 +522,11 @@ const thebe_selector_output = ".output, .cell_output"
|
||||
Cumulative gain curve
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#other-measures-in-classification-studies-cancer-data-again">
|
||||
Other measures in classification studies: Cancer Data again
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#material-for-lecture-thursday-november-9">
|
||||
Material for Lecture Thursday November 9
|
||||
@@ -664,8 +692,31 @@ const thebe_selector_output = ".output, .cell_output"
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#other-measures-in-classification-studies-cancer-data-again">
|
||||
Other measures in classification studies: Cancer Data again
|
||||
<a class="reference internal nav-link" href="#other-ways-of-presenting-a-classification-problem">
|
||||
Other ways of presenting a classification problem
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#combinations-of-classification-results">
|
||||
Combinations of classification results
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#positive-and-negative-prediction-values">
|
||||
Positive and negative prediction values
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#other-quantities">
|
||||
Other quantities
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#f-1-score">
|
||||
<span class="math notranslate nohighlight">
|
||||
\(F_1\)
|
||||
</span>
|
||||
score
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
@@ -678,6 +729,11 @@ const thebe_selector_output = ".output, .cell_output"
|
||||
Cumulative gain curve
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#other-measures-in-classification-studies-cancer-data-again">
|
||||
Other measures in classification studies: Cancer Data again
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#material-for-lecture-thursday-november-9">
|
||||
Material for Lecture Thursday November 9
|
||||
@@ -1019,7 +1075,7 @@ case under study.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>RandomizedSearchCV(estimator=Ridge(), n_iter=100,
|
||||
param_distributions={'alpha': <scipy.stats._distn_infrastructure.rv_frozen object at 0x105b84cd0>})
|
||||
param_distributions={'alpha': <scipy.stats._distn_infrastructure.rv_frozen object at 0x13d4a1640>})
|
||||
Best estimated lambda-value: 0.9849967686928113
|
||||
MSE score: 1.0853136633465326
|
||||
R2 score: -0.0002382102844775691
|
||||
@@ -1155,6 +1211,130 @@ features are of relevance and which are not. This leads us to
|
||||
the classical Principal Component Analysis (PCA) theorem with
|
||||
applications. This will be discussed later this semester (<a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/pub/week43/html/week43-bs.html">week 43</a>).</p>
|
||||
</div>
|
||||
<div class="section" id="other-ways-of-presenting-a-classification-problem">
|
||||
<h2>Other ways of presenting a classification problem<a class="headerlink" href="#other-ways-of-presenting-a-classification-problem" title="Permalink to this headline">¶</a></h2>
|
||||
<p>For a binary classifcation matrix, the so-called <strong>confusion matrix</strong>, is often used. It can also be extended to more catgeories/classes as well.
|
||||
The following quantities are then used</p>
|
||||
<ol class="simple">
|
||||
<li><p>positive condition number <span class="math notranslate nohighlight">\(P\)</span>, which represents the number of real positive cases in the data (output one/true etc)</p></li>
|
||||
<li><p>The condition negative number <span class="math notranslate nohighlight">\(N\)</span> which is the number of negative cases (ouput zero/false etc)</p></li>
|
||||
<li><p>The true positive number <span class="math notranslate nohighlight">\(TP\)</span> which represents whether a positive test result has been correctly classified (the application of our trained model on a test data set)</p></li>
|
||||
<li><p>The true negative <span class="math notranslate nohighlight">\(TN\)</span> number which represents whether a negative test has been correctly classified</p></li>
|
||||
<li><p>The false positive <span class="math notranslate nohighlight">\(FP\)</span> number, a so-called type I error which tells us about the fraction of positive test result which are wrongly classified</p></li>
|
||||
<li><p>A false negative <span class="math notranslate nohighlight">\(FN\)</span> number, a so-called type II error which, should be pretty obvious, indicates if a negative test has been wrongly classified.</p></li>
|
||||
</ol>
|
||||
<p>It is is easy to think in terms of illness. You could think of the above as</p>
|
||||
<ol class="simple">
|
||||
<li><p>True positive: Sick people correctly identified as sick</p></li>
|
||||
<li><p>False positive: Healthy people incorrectly identified as sick</p></li>
|
||||
<li><p>True negative: Healthy people correctly identified as healthy</p></li>
|
||||
<li><p>False negative: Sick people incorrectly identified as healthy</p></li>
|
||||
</ol>
|
||||
</div>
|
||||
<div class="section" id="combinations-of-classification-results">
|
||||
<h2>Combinations of classification results<a class="headerlink" href="#combinations-of-classification-results" title="Permalink to this headline">¶</a></h2>
|
||||
<p>It is common in the literature to define various combinations the above numbers. The most commonly used are</p>
|
||||
<p><strong>Sensitivity, recall, hit rate, or true positive rate <span class="math notranslate nohighlight">\(TPR\)</span>. It is the probability of a positive test result, conditioned on the individual truly being positive.</strong></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {TPR} ={\frac {\mathrm {TP} }{\mathrm {P} }}={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} }}=1-\mathrm {FNR} }
|
||||
\]</div>
|
||||
<p>The <span class="math notranslate nohighlight">\(TPR\)</span> defines how many correct positive results occur among all positive samples available during the test</p>
|
||||
<p><strong>Miss rate or false negative rate <span class="math notranslate nohighlight">\(FNR\)</span>.</strong></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {FNR} ={\frac {\mathrm {FN} }{\mathrm {P} }}={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TP} }} }
|
||||
\]</div>
|
||||
<p><strong>Specificity, selectivity or true negative rate <span class="math notranslate nohighlight">\(TNR\)</span>. It is the probability of a negative test result, conditioned on the individual truly being negative.</strong></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {TNR} ={\frac {\mathrm {TN} }{\mathrm {N} }}={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FP} }}=1-\mathrm {FPR} }
|
||||
\]</div>
|
||||
<p>with the fall-out false positive rate</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {FPR} ={\frac {\mathrm {FP} }{\mathrm {N} }}={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TN} }}=1-\mathrm {TNR} }
|
||||
\]</div>
|
||||
<p>The <span class="math notranslate nohighlight">\(FPR\)</span> defines how many incorrect positive results occur among
|
||||
all negative samples available during the test.</p>
|
||||
</div>
|
||||
<div class="section" id="positive-and-negative-prediction-values">
|
||||
<h2>Positive and negative prediction values<a class="headerlink" href="#positive-and-negative-prediction-values" title="Permalink to this headline">¶</a></h2>
|
||||
<p>The positive and negative predictive values
|
||||
are the proportions of positive and negative results in statistics and
|
||||
diagnostic tests that are true positive and true negative results,
|
||||
respectively.[1] The PPV and NPV describe the performance of a
|
||||
diagnostic test or other statistical measure. A high result can be
|
||||
interpreted as indicating the accuracy of such a statistic.</p>
|
||||
<p><strong>Precision or positive predictive value <span class="math notranslate nohighlight">\(PPV\)</span>.</strong></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {PPV} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FP} }}=1-\mathrm {FDR} }
|
||||
\]</div>
|
||||
<p><strong>Negative predictive value <span class="math notranslate nohighlight">\(NPV\)</span>.</strong></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {NPV} ={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FN} }}=1-\mathrm {FOR} }
|
||||
\]</div>
|
||||
</div>
|
||||
<div class="section" id="other-quantities">
|
||||
<h2>Other quantities<a class="headerlink" href="#other-quantities" title="Permalink to this headline">¶</a></h2>
|
||||
<p><strong>False discovery rate <span class="math notranslate nohighlight">\(FDR\)</span>.</strong></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {FDR} ={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TP} }}=1-\mathrm {PPV} }
|
||||
\]</div>
|
||||
<p><strong>False omission rate <span class="math notranslate nohighlight">\(FOR\)</span>.</strong></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {FOR} ={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TN} }}=1-\mathrm {NPV} }
|
||||
\]</div>
|
||||
</div>
|
||||
<div class="section" id="f-1-score">
|
||||
<h2><span class="math notranslate nohighlight">\(F_1\)</span> score<a class="headerlink" href="#f-1-score" title="Permalink to this headline">¶</a></h2>
|
||||
<p>In statistical analysis of binary classification, the F-score or
|
||||
F-measure is a measure of a test’s accuracy. It is calculated from the
|
||||
precision and recall of the test, where the precision is the number of
|
||||
true positive results divided by the number of all positive results,
|
||||
including those not identified correctly, and the recall is the number
|
||||
of true positive results divided by the number of all samples that
|
||||
should have been identified as positive. Precision is also known as
|
||||
positive predictive value, and recall is also known as sensitivity in
|
||||
diagnostic binary classification.</p>
|
||||
<p>The F1 score is the harmonic mean of the precision and recall. It thus
|
||||
symmetrically represents both precision and recall in one metric. The
|
||||
highest possible value of an F-score is 1.0, indicating perfect
|
||||
precision and recall, and the lowest possible value is 0, if either
|
||||
precision or recall are zero.</p>
|
||||
<p>It is defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \mathrm {F} _{1}=2\times {\frac {\mathrm {PPV} \times \mathrm {TPR} }{\mathrm {PPV} +\mathrm {TPR} }}={\frac {2\mathrm {TP} }{2\mathrm {TP} +\mathrm {FP} +\mathrm {FN} }}}
|
||||
\]</div>
|
||||
</div>
|
||||
<div class="section" id="roc-curve">
|
||||
<h2>ROC curve<a class="headerlink" href="#roc-curve" title="Permalink to this headline">¶</a></h2>
|
||||
<p>A receiver operating characteristic curve, or ROC curve, is a
|
||||
graphical plot that illustrates the performance of a binary classifier
|
||||
model at varying threshold values.</p>
|
||||
<p>The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.</p>
|
||||
<p>To draw a ROC curve, only the true positive rate (TPR) and false
|
||||
positive rate (FPR) are needed (as functions of some classifier
|
||||
parameter). The TPR defines how many correct positive results occur
|
||||
among all positive samples available during the test. FPR, on the
|
||||
other hand, defines how many incorrect positive results occur among
|
||||
all negative samples available during the test.</p>
|
||||
<p>See <a class="reference external" href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">https://en.wikipedia.org/wiki/Receiver_operating_characteristic</a> for more discussions.</p>
|
||||
</div>
|
||||
<div class="section" id="cumulative-gain-curve">
|
||||
<h2>Cumulative gain curve<a class="headerlink" href="#cumulative-gain-curve" title="Permalink to this headline">¶</a></h2>
|
||||
<p>The cumulative gain curve is a performance evaluation used typically for binary classification problems.
|
||||
It plots the <span class="math notranslate nohighlight">\(TPR\)</span> True Positive Rate or Sensitivity (which represents the
|
||||
fraction of examples correctly classified
|
||||
against Predictive Positive Rate, which represents
|
||||
the fraction of positively predicted examples.</p>
|
||||
<p>The examples below show the confusion matrix, the ROC curve and the cumulative gain for the Wisconsin cancer data.</p>
|
||||
</div>
|
||||
<div class="section" id="other-measures-in-classification-studies-cancer-data-again">
|
||||
<h2>Other measures in classification studies: Cancer Data again<a class="headerlink" href="#other-measures-in-classification-studies-cancer-data-again" title="Permalink to this headline">¶</a></h2>
|
||||
<div class="cell docutils container">
|
||||
@@ -1197,6 +1377,9 @@ applications. This will be discussed later this semester (<a class="reference ex
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(426, 30)
|
||||
(143, 30)
|
||||
[1. 0.86666667 1. 0.92857143 1. 0.85714286
|
||||
1. 0.92857143 0.92857143 1. ]
|
||||
Test set accuracy with Logistic Regression: 0.94
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
|
||||
@@ -1289,33 +1472,12 @@ Please also refer to the documentation for alternative solver options:
|
||||
n_iter_i = _check_optimize_result(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[1. 0.86666667 1. 0.92857143 1. 0.85714286
|
||||
1. 0.92857143 0.92857143 1. ]
|
||||
Test set accuracy with Logistic Regression: 0.94
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/week45_22_3.png" src="_images/week45_22_3.png" />
|
||||
<img alt="_images/week45_22_4.png" src="_images/week45_22_4.png" />
|
||||
<img alt="_images/week45_22_5.png" src="_images/week45_22_5.png" />
|
||||
<img alt="_images/week45_44_2.png" src="_images/week45_44_2.png" />
|
||||
<img alt="_images/week45_44_3.png" src="_images/week45_44_3.png" />
|
||||
<img alt="_images/week45_44_4.png" src="_images/week45_44_4.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="roc-curve">
|
||||
<h2>ROC curve<a class="headerlink" href="#roc-curve" title="Permalink to this headline">¶</a></h2>
|
||||
<p>A receiver operating characteristic curve, or ROC curve, is a
|
||||
graphical plot that illustrates the performance of a binary classifier
|
||||
model at varying threshold values.</p>
|
||||
<p>The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.
|
||||
See <a class="reference external" href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">https://en.wikipedia.org/wiki/Receiver_operating_characteristic</a> for more discussions.</p>
|
||||
</div>
|
||||
<div class="section" id="cumulative-gain-curve">
|
||||
<h2>Cumulative gain curve<a class="headerlink" href="#cumulative-gain-curve" title="Permalink to this headline">¶</a></h2>
|
||||
<p>The cumulative gain curve is a performance evaluation used typically for binary classification problems.
|
||||
It plots the <span class="math notranslate nohighlight">\(TPR\)</span> True Positive Rate or Sensitivity (which represents the
|
||||
fraction of examples correctly classified
|
||||
against Predictive Positive Rate, which represents
|
||||
the fraction of positively predicted examples.</p>
|
||||
</div>
|
||||
<div class="section" id="material-for-lecture-thursday-november-9">
|
||||
<h2>Material for Lecture Thursday November 9<a class="headerlink" href="#material-for-lecture-thursday-november-9" title="Permalink to this headline">¶</a></h2>
|
||||
</div>
|
||||
@@ -1457,285 +1619,303 @@ systems such as automatic translation and speech-to-text.</p>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 1/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2023-11-07 06:16:59.648382: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2023-11-08 06:51:57.259901: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 3s - loss: 0.9885 - 3s/epoch - 66ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 3s - loss: 2.5680 - 3s/epoch - 69ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 2/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4316 - 465ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 1.8934 - 563ms/epoch - 11ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 3/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4150 - 494ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 1.4588 - 512ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 4/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4066 - 466ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.8688 - 466ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 5/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4037 - 497ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4871 - 457ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 6/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4013 - 469ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4162 - 456ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 7/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3980 - 467ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4068 - 451ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 8/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3981 - 468ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4032 - 453ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 9/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3944 - 467ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4009 - 453ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 10/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3951 - 468ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3978 - 457ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 11/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3929 - 467ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3964 - 451ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 12/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3929 - 467ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3936 - 455ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 13/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3915 - 469ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3933 - 450ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 14/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3923 - 472ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3938 - 454ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 15/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3901 - 467ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3924 - 452ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 16/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3893 - 471ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3924 - 456ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 17/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3893 - 468ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3919 - 454ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 18/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3893 - 490ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3907 - 454ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 19/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3888 - 475ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3906 - 453ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 20/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3866 - 468ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3907 - 452ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 21/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3875 - 469ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3888 - 453ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 22/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3864 - 472ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3898 - 454ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 23/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3883 - 467ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3889 - 453ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 24/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3877 - 469ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3883 - 452ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 25/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3884 - 491ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3885 - 456ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 26/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3862 - 472ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3886 - 452ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 27/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3844 - 466ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3885 - 457ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 28/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3869 - 468ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3872 - 454ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 29/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3859 - 470ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3885 - 452ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 30/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3849 - 466ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3869 - 453ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 31/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3861 - 463ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3869 - 454ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 32/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3837 - 462ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3868 - 455ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 33/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3844 - 477ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3869 - 453ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 34/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3842 - 479ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3872 - 455ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 35/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3825 - 476ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3856 - 481ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 36/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3849 - 477ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3853 - 454ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 37/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3826 - 463ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3859 - 483ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 38/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3816 - 466ms/epoch - 9ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3862 - 482ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 39/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 0.3843 - 1s/epoch - 25ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 0.3850 - 508ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 40/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 0.3821 - 522ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3841 - 494ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 41/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3825 - 486ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3848 - 473ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 42/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3822 - 480ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3850 - 479ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 43/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 0.3836 - 521ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3841 - 473ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 44/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 0.3827 - 566ms/epoch - 11ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3847 - 479ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 45/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 0.3815 - 504ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3810 - 463ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 46/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 0.3832 - 511ms/epoch - 10ms/step
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3843 - 483ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 47/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3830 - 467ms/epoch - 9ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 48/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3827 - 485ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 49/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3820 - 479ms/epoch - 10ms/step
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 50/100
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
<span class="ne">KeyboardInterrupt</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
|
||||
<span class="nn">Input In [9],</span> in <span class="ni"><cell line: 53></span><span class="nt">()</span>
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -334,6 +334,150 @@ correlation_matrix = cancerpd.corr().round(1)
|
||||
# the classical Principal Component Analysis (PCA) theorem with
|
||||
# applications. This will be discussed later this semester ([week 43](https://compphysics.github.io/MachineLearning/doc/pub/week43/html/week43-bs.html)).
|
||||
|
||||
# ## Other ways of presenting a classification problem
|
||||
#
|
||||
# For a binary classifcation matrix, the so-called **confusion matrix**, is often used. It can also be extended to more catgeories/classes as well.
|
||||
# The following quantities are then used
|
||||
# 1. positive condition number $P$, which represents the number of real positive cases in the data (output one/true etc)
|
||||
#
|
||||
# 2. The condition negative number $N$ which is the number of negative cases (ouput zero/false etc)
|
||||
#
|
||||
# 3. The true positive number $TP$ which represents whether a positive test result has been correctly classified (the application of our trained model on a test data set)
|
||||
#
|
||||
# 4. The true negative $TN$ number which represents whether a negative test has been correctly classified
|
||||
#
|
||||
# 5. The false positive $FP$ number, a so-called type I error which tells us about the fraction of positive test result which are wrongly classified
|
||||
#
|
||||
# 6. A false negative $FN$ number, a so-called type II error which, should be pretty obvious, indicates if a negative test has been wrongly classified.
|
||||
#
|
||||
# It is is easy to think in terms of illness. You could think of the above as
|
||||
# 1. True positive: Sick people correctly identified as sick
|
||||
#
|
||||
# 2. False positive: Healthy people incorrectly identified as sick
|
||||
#
|
||||
# 3. True negative: Healthy people correctly identified as healthy
|
||||
#
|
||||
# 4. False negative: Sick people incorrectly identified as healthy
|
||||
|
||||
# ## Combinations of classification results
|
||||
#
|
||||
# It is common in the literature to define various combinations the above numbers. The most commonly used are
|
||||
#
|
||||
# **Sensitivity, recall, hit rate, or true positive rate $TPR$. It is the probability of a positive test result, conditioned on the individual truly being positive.**
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {TPR} ={\frac {\mathrm {TP} }{\mathrm {P} }}={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} }}=1-\mathrm {FNR} }
|
||||
# $$
|
||||
|
||||
# The $TPR$ defines how many correct positive results occur among all positive samples available during the test
|
||||
#
|
||||
# **Miss rate or false negative rate $FNR$.**
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {FNR} ={\frac {\mathrm {FN} }{\mathrm {P} }}={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TP} }} }
|
||||
# $$
|
||||
|
||||
# **Specificity, selectivity or true negative rate $TNR$. It is the probability of a negative test result, conditioned on the individual truly being negative.**
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {TNR} ={\frac {\mathrm {TN} }{\mathrm {N} }}={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FP} }}=1-\mathrm {FPR} }
|
||||
# $$
|
||||
|
||||
# with the fall-out false positive rate
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {FPR} ={\frac {\mathrm {FP} }{\mathrm {N} }}={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TN} }}=1-\mathrm {TNR} }
|
||||
# $$
|
||||
|
||||
# The $FPR$ defines how many incorrect positive results occur among
|
||||
# all negative samples available during the test.
|
||||
|
||||
# ## Positive and negative prediction values
|
||||
#
|
||||
# The positive and negative predictive values
|
||||
# are the proportions of positive and negative results in statistics and
|
||||
# diagnostic tests that are true positive and true negative results,
|
||||
# respectively.[1] The PPV and NPV describe the performance of a
|
||||
# diagnostic test or other statistical measure. A high result can be
|
||||
# interpreted as indicating the accuracy of such a statistic.
|
||||
#
|
||||
# **Precision or positive predictive value $PPV$.**
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {PPV} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FP} }}=1-\mathrm {FDR} }
|
||||
# $$
|
||||
|
||||
# **Negative predictive value $NPV$.**
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {NPV} ={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FN} }}=1-\mathrm {FOR} }
|
||||
# $$
|
||||
|
||||
# ## Other quantities
|
||||
#
|
||||
# **False discovery rate $FDR$.**
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {FDR} ={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TP} }}=1-\mathrm {PPV} }
|
||||
# $$
|
||||
|
||||
# **False omission rate $FOR$.**
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {FOR} ={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TN} }}=1-\mathrm {NPV} }
|
||||
# $$
|
||||
|
||||
# ## $F_1$ score
|
||||
#
|
||||
# In statistical analysis of binary classification, the F-score or
|
||||
# F-measure is a measure of a test's accuracy. It is calculated from the
|
||||
# precision and recall of the test, where the precision is the number of
|
||||
# true positive results divided by the number of all positive results,
|
||||
# including those not identified correctly, and the recall is the number
|
||||
# of true positive results divided by the number of all samples that
|
||||
# should have been identified as positive. Precision is also known as
|
||||
# positive predictive value, and recall is also known as sensitivity in
|
||||
# diagnostic binary classification.
|
||||
#
|
||||
# The F1 score is the harmonic mean of the precision and recall. It thus
|
||||
# symmetrically represents both precision and recall in one metric. The
|
||||
# highest possible value of an F-score is 1.0, indicating perfect
|
||||
# precision and recall, and the lowest possible value is 0, if either
|
||||
# precision or recall are zero.
|
||||
#
|
||||
# It is defined as
|
||||
|
||||
# $$
|
||||
# {\displaystyle \mathrm {F} _{1}=2\times {\frac {\mathrm {PPV} \times \mathrm {TPR} }{\mathrm {PPV} +\mathrm {TPR} }}={\frac {2\mathrm {TP} }{2\mathrm {TP} +\mathrm {FP} +\mathrm {FN} }}}
|
||||
# $$
|
||||
|
||||
# ## ROC curve
|
||||
#
|
||||
# A receiver operating characteristic curve, or ROC curve, is a
|
||||
# graphical plot that illustrates the performance of a binary classifier
|
||||
# model at varying threshold values.
|
||||
#
|
||||
# The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.
|
||||
#
|
||||
# To draw a ROC curve, only the true positive rate (TPR) and false
|
||||
# positive rate (FPR) are needed (as functions of some classifier
|
||||
# parameter). The TPR defines how many correct positive results occur
|
||||
# among all positive samples available during the test. FPR, on the
|
||||
# other hand, defines how many incorrect positive results occur among
|
||||
# all negative samples available during the test.
|
||||
#
|
||||
# See <https://en.wikipedia.org/wiki/Receiver_operating_characteristic> for more discussions.
|
||||
|
||||
# ## Cumulative gain curve
|
||||
#
|
||||
# The cumulative gain curve is a performance evaluation used typically for binary classification problems.
|
||||
# It plots the $TPR$ True Positive Rate or Sensitivity (which represents the
|
||||
# fraction of examples correctly classified
|
||||
# against Predictive Positive Rate, which represents
|
||||
# the fraction of positively predicted examples.
|
||||
#
|
||||
# The examples below show the confusion matrix, the ROC curve and the cumulative gain for the Wisconsin cancer data.
|
||||
|
||||
# ## Other measures in classification studies: Cancer Data again
|
||||
|
||||
# In[8]:
|
||||
@@ -373,23 +517,6 @@ skplt.metrics.plot_cumulative_gain(y_test, y_probas)
|
||||
plt.show()
|
||||
|
||||
|
||||
# ## ROC curve
|
||||
#
|
||||
# A receiver operating characteristic curve, or ROC curve, is a
|
||||
# graphical plot that illustrates the performance of a binary classifier
|
||||
# model at varying threshold values.
|
||||
#
|
||||
# The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.
|
||||
# See <https://en.wikipedia.org/wiki/Receiver_operating_characteristic> for more discussions.
|
||||
|
||||
# ## Cumulative gain curve
|
||||
#
|
||||
# The cumulative gain curve is a performance evaluation used typically for binary classification problems.
|
||||
# It plots the $TPR$ True Positive Rate or Sensitivity (which represents the
|
||||
# fraction of examples correctly classified
|
||||
# against Predictive Positive Rate, which represents
|
||||
# the fraction of positively predicted examples.
|
||||
|
||||
# ## Material for Lecture Thursday November 9
|
||||
|
||||
# ## Recurrent neural networks (RNNs): Overarching view
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
+368
-81
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ddffe3be",
|
||||
"id": "31a10e2b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cde0bbff",
|
||||
"id": "019daa83",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f48c6388",
|
||||
"id": "6ecf1c2b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -67,7 +67,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ba6964ff",
|
||||
"id": "d80a4db8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -77,7 +77,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4d810bbf",
|
||||
"id": "1ded5c92",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -96,7 +96,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "39f540b5",
|
||||
"id": "6dea0513",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -153,7 +153,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "09d24c34",
|
||||
"id": "440bf579",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -165,7 +165,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7c84429f",
|
||||
"id": "683b1da2",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -180,7 +180,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "ac8fda39",
|
||||
"id": "38be79e0",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -233,7 +233,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b9da31b6",
|
||||
"id": "081fbe5c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -248,7 +248,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "afe2e276",
|
||||
"id": "a9798c07",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -268,7 +268,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "5707b319",
|
||||
"id": "0a7e4e2e",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -322,7 +322,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f067ea7",
|
||||
"id": "dcd07cb8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -337,7 +337,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "cfe2ed78",
|
||||
"id": "65061d95",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -364,7 +364,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "97a315dc",
|
||||
"id": "50492453",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -378,7 +378,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "9363662a",
|
||||
"id": "08a2ab17",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -423,7 +423,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "298800d9",
|
||||
"id": "5408b7dd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -448,7 +448,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "9db77f36",
|
||||
"id": "76c3d259",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -460,7 +460,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2872089d",
|
||||
"id": "83903231",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -471,7 +471,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "926cdfd7",
|
||||
"id": "58ccd6c3",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -483,7 +483,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6bd13984",
|
||||
"id": "adda54cb",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -496,7 +496,327 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "55242d6d",
|
||||
"id": "8799c034",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Other ways of presenting a classification problem\n",
|
||||
"\n",
|
||||
"For a binary classifcation matrix, the so-called **confusion matrix**, is often used. It can also be extended to more catgeories/classes as well.\n",
|
||||
"The following quantities are then used\n",
|
||||
"1. positive condition number $P$, which represents the number of real positive cases in the data (output one/true etc)\n",
|
||||
"\n",
|
||||
"2. The condition negative number $N$ which is the number of negative cases (ouput zero/false etc)\n",
|
||||
"\n",
|
||||
"3. The true positive number $TP$ which represents whether a positive test result has been correctly classified (the application of our trained model on a test data set)\n",
|
||||
"\n",
|
||||
"4. The true negative $TN$ number which represents whether a negative test has been correctly classified\n",
|
||||
"\n",
|
||||
"5. The false positive $FP$ number, a so-called type I error which tells us about the fraction of positive test result which are wrongly classified\n",
|
||||
"\n",
|
||||
"6. A false negative $FN$ number, a so-called type II error which, should be pretty obvious, indicates if a negative test has been wrongly classified.\n",
|
||||
"\n",
|
||||
"It is is easy to think in terms of illness. You could think of the above as\n",
|
||||
"1. True positive: Sick people correctly identified as sick\n",
|
||||
"\n",
|
||||
"2. False positive: Healthy people incorrectly identified as sick\n",
|
||||
"\n",
|
||||
"3. True negative: Healthy people correctly identified as healthy\n",
|
||||
"\n",
|
||||
"4. False negative: Sick people incorrectly identified as healthy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dfbe0571",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Combinations of classification results\n",
|
||||
"\n",
|
||||
"It is common in the literature to define various combinations the above numbers. The most commonly used are\n",
|
||||
"\n",
|
||||
"**Sensitivity, recall, hit rate, or true positive rate $TPR$. It is the probability of a positive test result, conditioned on the individual truly being positive.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "65f7c63d",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {TPR} ={\\frac {\\mathrm {TP} }{\\mathrm {P} }}={\\frac {\\mathrm {TP} }{\\mathrm {TP} +\\mathrm {FN} }}=1-\\mathrm {FNR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2e674d10",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"The $TPR$ defines how many correct positive results occur among all positive samples available during the test\n",
|
||||
"\n",
|
||||
"**Miss rate or false negative rate $FNR$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "65577837",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {FNR} ={\\frac {\\mathrm {FN} }{\\mathrm {P} }}={\\frac {\\mathrm {FN} }{\\mathrm {FN} +\\mathrm {TP} }} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e5e8802d",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**Specificity, selectivity or true negative rate $TNR$. It is the probability of a negative test result, conditioned on the individual truly being negative.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "07c61591",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {TNR} ={\\frac {\\mathrm {TN} }{\\mathrm {N} }}={\\frac {\\mathrm {TN} }{\\mathrm {TN} +\\mathrm {FP} }}=1-\\mathrm {FPR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b3f61bb3",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"with the fall-out false positive rate"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6544105e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {FPR} ={\\frac {\\mathrm {FP} }{\\mathrm {N} }}={\\frac {\\mathrm {FP} }{\\mathrm {FP} +\\mathrm {TN} }}=1-\\mathrm {TNR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0b571e85",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"The $FPR$ defines how many incorrect positive results occur among\n",
|
||||
"all negative samples available during the test."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "17d6872a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Positive and negative prediction values\n",
|
||||
"\n",
|
||||
"The positive and negative predictive values \n",
|
||||
"are the proportions of positive and negative results in statistics and\n",
|
||||
"diagnostic tests that are true positive and true negative results,\n",
|
||||
"respectively.[1] The PPV and NPV describe the performance of a\n",
|
||||
"diagnostic test or other statistical measure. A high result can be\n",
|
||||
"interpreted as indicating the accuracy of such a statistic.\n",
|
||||
"\n",
|
||||
"**Precision or positive predictive value $PPV$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "122f7e6c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {PPV} ={\\frac {\\mathrm {TP} }{\\mathrm {TP} +\\mathrm {FP} }}=1-\\mathrm {FDR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f44a5e78",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**Negative predictive value $NPV$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "86fa9670",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {NPV} ={\\frac {\\mathrm {TN} }{\\mathrm {TN} +\\mathrm {FN} }}=1-\\mathrm {FOR} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a30e358d",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Other quantities\n",
|
||||
"\n",
|
||||
"**False discovery rate $FDR$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3f0de6de",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {FDR} ={\\frac {\\mathrm {FP} }{\\mathrm {FP} +\\mathrm {TP} }}=1-\\mathrm {PPV} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0b4cc93f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**False omission rate $FOR$.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4866ec0a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {FOR} ={\\frac {\\mathrm {FN} }{\\mathrm {FN} +\\mathrm {TN} }}=1-\\mathrm {NPV} }\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9b1c8c45",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## $F_1$ score\n",
|
||||
"\n",
|
||||
"In statistical analysis of binary classification, the F-score or\n",
|
||||
"F-measure is a measure of a test's accuracy. It is calculated from the\n",
|
||||
"precision and recall of the test, where the precision is the number of\n",
|
||||
"true positive results divided by the number of all positive results,\n",
|
||||
"including those not identified correctly, and the recall is the number\n",
|
||||
"of true positive results divided by the number of all samples that\n",
|
||||
"should have been identified as positive. Precision is also known as\n",
|
||||
"positive predictive value, and recall is also known as sensitivity in\n",
|
||||
"diagnostic binary classification.\n",
|
||||
"\n",
|
||||
"The F1 score is the harmonic mean of the precision and recall. It thus\n",
|
||||
"symmetrically represents both precision and recall in one metric. The\n",
|
||||
"highest possible value of an F-score is 1.0, indicating perfect\n",
|
||||
"precision and recall, and the lowest possible value is 0, if either\n",
|
||||
"precision or recall are zero.\n",
|
||||
"\n",
|
||||
"It is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0e651a1f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"{\\displaystyle \\mathrm {F} _{1}=2\\times {\\frac {\\mathrm {PPV} \\times \\mathrm {TPR} }{\\mathrm {PPV} +\\mathrm {TPR} }}={\\frac {2\\mathrm {TP} }{2\\mathrm {TP} +\\mathrm {FP} +\\mathrm {FN} }}}\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d83efe41",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## ROC curve\n",
|
||||
"\n",
|
||||
"A receiver operating characteristic curve, or ROC curve, is a\n",
|
||||
"graphical plot that illustrates the performance of a binary classifier\n",
|
||||
"model at varying threshold values.\n",
|
||||
"\n",
|
||||
"The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.\n",
|
||||
"\n",
|
||||
"To draw a ROC curve, only the true positive rate (TPR) and false\n",
|
||||
"positive rate (FPR) are needed (as functions of some classifier\n",
|
||||
"parameter). The TPR defines how many correct positive results occur\n",
|
||||
"among all positive samples available during the test. FPR, on the\n",
|
||||
"other hand, defines how many incorrect positive results occur among\n",
|
||||
"all negative samples available during the test.\n",
|
||||
"\n",
|
||||
"See <https://en.wikipedia.org/wiki/Receiver_operating_characteristic> for more discussions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d088215b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Cumulative gain curve\n",
|
||||
"\n",
|
||||
"The cumulative gain curve is a performance evaluation used typically for binary classification problems.\n",
|
||||
"It plots the $TPR$ True Positive Rate or Sensitivity (which represents the \n",
|
||||
"fraction of examples correctly classified\n",
|
||||
"against Predictive Positive Rate, which represents \n",
|
||||
"the fraction of positively predicted examples.\n",
|
||||
"\n",
|
||||
"The examples below show the confusion matrix, the ROC curve and the cumulative gain for the Wisconsin cancer data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5c0bcd1f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -507,7 +827,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "00c15782",
|
||||
"id": "2a6f80df",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -550,40 +870,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7a8eeb7",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## ROC curve\n",
|
||||
"\n",
|
||||
"A receiver operating characteristic curve, or ROC curve, is a\n",
|
||||
"graphical plot that illustrates the performance of a binary classifier\n",
|
||||
"model at varying threshold values.\n",
|
||||
"\n",
|
||||
"The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.\n",
|
||||
"See <https://en.wikipedia.org/wiki/Receiver_operating_characteristic> for more discussions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6414e652",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Cumulative gain curve\n",
|
||||
"\n",
|
||||
"The cumulative gain curve is a performance evaluation used typically for binary classification problems.\n",
|
||||
"It plots the $TPR$ True Positive Rate or Sensitivity (which represents the \n",
|
||||
"fraction of examples correctly classified\n",
|
||||
"against Predictive Positive Rate, which represents \n",
|
||||
"the fraction of positively predicted examples."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7646c025",
|
||||
"id": "58e5b521",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -593,7 +880,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f7b4b23b",
|
||||
"id": "4ebe5a91",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -620,7 +907,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "741bf972",
|
||||
"id": "182f425e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -631,7 +918,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "f9060416",
|
||||
"id": "8b3ca785",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -704,7 +991,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e0342b0b",
|
||||
"id": "b0c12b4c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -724,7 +1011,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "58fdc710",
|
||||
"id": "d81020f6",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -740,7 +1027,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f793c129",
|
||||
"id": "2d7453c9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -762,7 +1049,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7446349",
|
||||
"id": "fc4a082e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -780,7 +1067,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fac85e06",
|
||||
"id": "c1106ad9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -826,7 +1113,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7e3fbdb7",
|
||||
"id": "b139ef7b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -847,7 +1134,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f29008fb",
|
||||
"id": "70c95078",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -908,7 +1195,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dedfb5dd",
|
||||
"id": "2ca24031",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -933,7 +1220,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2abfa20e",
|
||||
"id": "93648979",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -952,7 +1239,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "abc4dabb",
|
||||
"id": "b8806dcd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -976,7 +1263,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "67e1ca86",
|
||||
"id": "b0edad40",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1058,7 +1345,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bc04ebe9",
|
||||
"id": "58fd18f9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1074,7 +1361,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "a5882cbb",
|
||||
"id": "a604caa5",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1113,7 +1400,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "18221ec4",
|
||||
"id": "545336c9",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1157,7 +1444,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "d4982f3a",
|
||||
"id": "3ef3ed13",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1240,7 +1527,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a146c2c5",
|
||||
"id": "8a8b29b8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1251,7 +1538,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "2f7691d6",
|
||||
"id": "d840beea",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1353,7 +1640,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "618a775c",
|
||||
"id": "2e5b339f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1375,7 +1662,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "5f955304",
|
||||
"id": "370e799d",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1472,7 +1759,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ae8bb1a0",
|
||||
"id": "d8f62fc4",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1498,7 +1785,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "701451d4",
|
||||
"id": "4ea36ea9",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
|
||||
Reference in New Issue
Block a user