diff --git a/doc/pub/week44/ipynb/week44.ipynb b/doc/pub/week44/ipynb/week44.ipynb index 1e94b3de9..4f6d3e187 100644 --- a/doc/pub/week44/ipynb/week44.ipynb +++ b/doc/pub/week44/ipynb/week44.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "bf6a66bd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "7af436e6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 44: Dimensionality Reduction, PCA and Clustering. Decision Trees\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -30,9 +26,7 @@ { "cell_type": "markdown", "id": "98ddd7e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Overview of week 44\n", "\n", @@ -58,9 +52,7 @@ { "cell_type": "markdown", "id": "70dec958", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Thursday, Principal Component Analysis\n", "\n", @@ -71,9 +63,7 @@ { "cell_type": "markdown", "id": "99b145fd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Thursday: Clustering and Unsupervised Learning\n", "\n", @@ -91,9 +81,7 @@ { "cell_type": "markdown", "id": "ac970c3d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Basic Idea of the $k$-means Clustering Algorithm\n", "\n", @@ -107,9 +95,7 @@ { "cell_type": "markdown", "id": "b70e75a5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The $k$-means Algorithm\n", "\n", @@ -120,9 +106,7 @@ { "cell_type": "markdown", "id": "ecee5684", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "k\\in\\{1, \\cdots, K \\}$.\n", @@ -132,9 +116,7 @@ { "cell_type": "markdown", "id": "64b830ad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the basic k-means algorithm each point is assigned to only\n", "one cluster $k$, and these assignments are *non-injective* i.e. many-to-one. We\n", @@ -154,9 +136,7 @@ { "cell_type": "markdown", "id": "99c98635", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Basic Math of the $k$-means Algorithm\n", "\n", @@ -166,9 +146,7 @@ { "cell_type": "markdown", "id": "785493cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -183,9 +161,7 @@ { "cell_type": "markdown", "id": "2a163123", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we wish to group into $K < n$ clusters. For our dissimilarity measure we\n", "use the *squared Euclidean distance*" @@ -194,9 +170,7 @@ { "cell_type": "markdown", "id": "7116fb2f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -212,9 +186,7 @@ { "cell_type": "markdown", "id": "fc62645d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Within Cluster Point Scatter\n", "\n", @@ -226,9 +198,7 @@ { "cell_type": "markdown", "id": "383635b6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -245,9 +215,7 @@ { "cell_type": "markdown", "id": "e085595c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\overline{x_k}}$ is the mean vector associated with the $k$-th\n", "cluster, and $N_k = \\sum_{i=1}^nI(C(i) = k)$, where the $I()$ notation is\n", @@ -262,9 +230,7 @@ { "cell_type": "markdown", "id": "1637663e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More Details\n", "\n", @@ -274,9 +240,7 @@ { "cell_type": "markdown", "id": "9c0e866b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -295,9 +259,7 @@ { "cell_type": "markdown", "id": "06ad9c98", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This is a quantity that is conserved throughout the $k$-means algorithm. It can\n", "be thought of as the total amount of information in the data, and it is composed\n", @@ -309,9 +271,7 @@ { "cell_type": "markdown", "id": "1ab25d0c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Total Cluster Variance\n", "Given a cluster mean $\\boldsymbol{m_k}$ we define the **total cluster variance**" @@ -320,9 +280,7 @@ { "cell_type": "markdown", "id": "0bbf889e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -337,9 +295,7 @@ { "cell_type": "markdown", "id": "92571f86", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Now we have all the pieces necessary to formally revisit the $k$-means algorithm." ] @@ -347,9 +303,7 @@ { "cell_type": "markdown", "id": "ce2da1a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The $k$-means Clustering Algorithm\n", "\n", @@ -365,9 +319,7 @@ { "cell_type": "markdown", "id": "bc6b373b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Summarizing\n", "\n", @@ -385,9 +337,7 @@ { "cell_type": "markdown", "id": "918ffb62", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Writing our own Code, the Data Set\n", "\n", @@ -405,11 +355,20 @@ "cell_type": "code", "execution_count": 1, "id": "f3e96eb4", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "ename": "ModuleNotFoundError", + "evalue": "No module named 'tensorflow'", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mtensorflow\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mimage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpyplot\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'tensorflow'" + ] + } + ], "source": [ "%matplotlib inline\n", "\n", @@ -427,9 +386,7 @@ { "cell_type": "markdown", "id": "2c23e020", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Next we define functions, for ease of use later, to generate Gaussians and to\n", "set up our toy data set." @@ -439,11 +396,21 @@ "cell_type": "code", "execution_count": 2, "id": "6d00f953", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "ename": "NameError", + "evalue": "name 'plt' is not defined", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 47\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 48\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 49\u001b[0;31m \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgenerate_simple_clustering_dataset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;32m\u001b[0m in \u001b[0;36mgenerate_simple_clustering_dataset\u001b[0;34m(dim, n_points, plotting, return_data)\u001b[0m\n\u001b[1;32m 37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 38\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mplotting\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 39\u001b[0;31m \u001b[0mfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0max\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubplots\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 40\u001b[0m \u001b[0max\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscatter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 41\u001b[0m \u001b[0max\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_title\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Toy Model Dataset'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mNameError\u001b[0m: name 'plt' is not defined" + ] + } + ], "source": [ "def gaussian_points(dim=2, n_points=1000, mean_vector=np.array([0, 0]),\n", " sample_variance=1):\n", @@ -499,9 +466,7 @@ { "cell_type": "markdown", "id": "a12ceb78", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Implementing the $k$-means Algorithm\n", "\n", @@ -513,10 +478,7 @@ "cell_type": "code", "execution_count": 3, "id": "576200d6", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -558,9 +520,7 @@ { "cell_type": "markdown", "id": "1df6f1a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plotting" ] @@ -569,10 +529,7 @@ "cell_type": "code", "execution_count": 4, "id": "2cb7e75b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "fig = plt.figure()\n", @@ -593,9 +550,7 @@ { "cell_type": "markdown", "id": "5fa3b389", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "So what do we have so far? We have 'picked' $k$ centroids at random from our\n", "data points. There are other ways of more intelligently choosing their\n", @@ -612,9 +567,7 @@ { "cell_type": "markdown", "id": "6a0d93e3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Continuing" ] @@ -623,10 +576,7 @@ "cell_type": "code", "execution_count": 5, "id": "7befecf7", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -679,9 +629,7 @@ { "cell_type": "markdown", "id": "6db65597", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Wrapping it up\n", "We now have a simple , un-optimized $k$-means\n", @@ -692,10 +640,7 @@ "cell_type": "code", "execution_count": 6, "id": "c7f59758", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "fig = plt.figure()\n", @@ -717,10 +662,7 @@ "cell_type": "code", "execution_count": 7, "id": "6434c06d", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "def naive_kmeans(data, n_clusters=4, max_iterations=100, tolerance=1e-8):\n", @@ -796,9 +738,7 @@ { "cell_type": "markdown", "id": "2e3a547a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Decision trees, overarching aims\n", "\n", @@ -827,9 +767,7 @@ { "cell_type": "markdown", "id": "8026fa4f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Basics of a tree\n", "\n", @@ -847,9 +785,7 @@ { "cell_type": "markdown", "id": "9e1373c9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A Sketch of a Tree, Regression problem\n", "\n", @@ -859,9 +795,7 @@ { "cell_type": "markdown", "id": "bbc6c594", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A Sketch of a Tree, Classification problem\n", "\n", @@ -871,9 +805,7 @@ { "cell_type": "markdown", "id": "30e044f6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A typical Decision Tree with its pertinent Jargon, Classification Problem\n", "\n", @@ -889,9 +821,7 @@ { "cell_type": "markdown", "id": "14eb6577", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## General Features\n", "\n", @@ -912,9 +842,7 @@ { "cell_type": "markdown", "id": "52d0f1f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## How do we set it up?\n", "\n", @@ -935,9 +863,7 @@ { "cell_type": "markdown", "id": "f4fcf249", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Decision trees and Regression" ] @@ -946,10 +872,7 @@ "cell_type": "code", "execution_count": 8, "id": "b81178c5", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1045,9 +968,7 @@ { "cell_type": "markdown", "id": "f1407bd2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Building a tree, regression\n", "\n", @@ -1067,9 +988,7 @@ { "cell_type": "markdown", "id": "e159978f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{j=1}^J\\sum_{i\\in R_j}(y_i-\\overline{y}_{R_j})^2,\n", @@ -1079,9 +998,7 @@ { "cell_type": "markdown", "id": "ac02d4ca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\overline{y}_{R_j}$ is the mean response for the training observations \n", "within box $j$." @@ -1090,9 +1007,7 @@ { "cell_type": "markdown", "id": "d0fa609a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A top-down approach, recursive binary splitting\n", "\n", @@ -1112,9 +1027,7 @@ { "cell_type": "markdown", "id": "e823cca8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Making a tree\n", "\n", @@ -1125,9 +1038,7 @@ { "cell_type": "markdown", "id": "a8c7a43a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left\\{X\\vert x_j < s\\right\\},\n", @@ -1137,9 +1048,7 @@ { "cell_type": "markdown", "id": "13fd8d8b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1147,9 +1056,7 @@ { "cell_type": "markdown", "id": "fd034108", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left\\{X\\vert x_j \\geq s\\right\\},\n", @@ -1159,9 +1066,7 @@ { "cell_type": "markdown", "id": "c6913579", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "so that we obtain the lowest MSE, that is" ] @@ -1169,9 +1074,7 @@ { "cell_type": "markdown", "id": "c9259557", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{i:x_i\\in R_j}(y_i-\\overline{y}_{R_1})^2+\\sum_{i:x_i\\in R_2}(y_i-\\overline{y}_{R_2})^2,\n", @@ -1181,9 +1084,7 @@ { "cell_type": "markdown", "id": "292547d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we want to minimize by considering all predictors\n", "$x_1,x_2,\\dots,x_p$. We consider also all possible values of $s$ for\n", @@ -1214,9 +1115,7 @@ { "cell_type": "markdown", "id": "01e8457e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Pruning the tree\n", "\n", @@ -1238,9 +1137,7 @@ { "cell_type": "markdown", "id": "08d0bb4a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Cost complexity pruning\n", "\n", @@ -1250,9 +1147,7 @@ { "cell_type": "markdown", "id": "28bb2119", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{m=1}^{\\overline{T}}\\sum_{i:x_i\\in R_m}(y_i-\\overline{y}_{R_m})^2+\\alpha\\overline{T},\n", @@ -1262,9 +1157,7 @@ { "cell_type": "markdown", "id": "015793a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "is as small as possible. Here $\\overline{T}$ is \n", "the number of terminal nodes of the tree $T$ , $R_m$ is the\n", @@ -1290,9 +1183,7 @@ { "cell_type": "markdown", "id": "92c92e1f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Schematic Regression Procedure\n", "\n", @@ -1316,9 +1207,7 @@ { "cell_type": "markdown", "id": "44631e32", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A Classification Tree\n", "\n", @@ -1339,9 +1228,7 @@ { "cell_type": "markdown", "id": "ff27eba7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Growing a classification tree\n", "\n", @@ -1366,9 +1253,7 @@ { "cell_type": "markdown", "id": "ed03abb4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Classification tree, how to split nodes\n", "\n", @@ -1385,9 +1270,7 @@ { "cell_type": "markdown", "id": "2c8d81e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p_{mk} = \\frac{1}{N_m}\\sum_{x_i\\in R_m}I(y_i=k).\n", @@ -1397,9 +1280,7 @@ { "cell_type": "markdown", "id": "3c3785a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We let $p_{mk}$ represent the majority class of observations in region\n", "$m$. The three most common ways of splitting a node are given by\n", @@ -1410,9 +1291,7 @@ { "cell_type": "markdown", "id": "b479e704", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p_{mk} = \\frac{1}{N_m}\\sum_{x_i\\in R_m}I(y_i\\ne k) = 1-p_{mk}.\n", @@ -1422,9 +1301,7 @@ { "cell_type": "markdown", "id": "a7edcc9e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "* Gini index $g$" ] @@ -1432,9 +1309,7 @@ { "cell_type": "markdown", "id": "9f60e695", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "g = \\sum_{k=1}^K p_{mk}(1-p_{mk}).\n", @@ -1444,9 +1319,7 @@ { "cell_type": "markdown", "id": "a5260cc5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "* Information entropy or just entropy $s$" ] @@ -1454,9 +1327,7 @@ { "cell_type": "markdown", "id": "9c5a41b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "s = -\\sum_{k=1}^K p_{mk}\\log{p_{mk}}.\n", @@ -1466,9 +1337,7 @@ { "cell_type": "markdown", "id": "7d290873", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Visualizing the Tree, Classification" ] @@ -1477,10 +1346,7 @@ "cell_type": "code", "execution_count": 9, "id": "acf9d5e4", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import os\n", @@ -1521,9 +1387,7 @@ { "cell_type": "markdown", "id": "ba61689c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Visualizing the Tree, The Moons" ] @@ -1532,10 +1396,7 @@ "cell_type": "code", "execution_count": 10, "id": "52e2eeb3", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Common imports\n", @@ -1567,9 +1428,7 @@ { "cell_type": "markdown", "id": "ff909dae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Other ways of visualizing the trees\n", "\n", @@ -1580,10 +1439,7 @@ "cell_type": "code", "execution_count": 11, "id": "87ee9247", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_iris\n", @@ -1598,9 +1454,7 @@ { "cell_type": "markdown", "id": "a6f7e44c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Printing out as text\n", "\n", @@ -1612,10 +1466,7 @@ "cell_type": "code", "execution_count": 12, "id": "a7f81bcd", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_iris\n", @@ -1631,9 +1482,7 @@ { "cell_type": "markdown", "id": "e4f6ba35", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Algorithms for Setting up Decision Trees\n", "\n", @@ -1651,9 +1500,7 @@ { "cell_type": "markdown", "id": "4c79a12f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The CART algorithm for Classification\n", "\n", @@ -1668,9 +1515,7 @@ { "cell_type": "markdown", "id": "7825f2c9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(k,t_k) = \\frac{m_{\\mathrm{left}}}{m}G_{\\mathrm{left}}+ \\frac{m_{\\mathrm{right}}}{m}G_{\\mathrm{right}},\n", @@ -1680,9 +1525,7 @@ { "cell_type": "markdown", "id": "0b432770", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $G_{\\mathrm{left/right}}$ measures the impurity of the left/right subset and $m_{\\mathrm{left/right}}$\n", " is the number of instances in the left/right subset\n", @@ -1697,9 +1540,7 @@ { "cell_type": "markdown", "id": "ce71bcb8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The CART algorithm for Regression\n", "\n", @@ -1710,9 +1551,7 @@ { "cell_type": "markdown", "id": "e54a3968", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(k,t_k) = \\frac{m_{\\mathrm{left}}}{m}\\mathrm{MSE}_{\\mathrm{left}}+ \\frac{m_{\\mathrm{right}}}{m}\\mathrm{MSE}_{\\mathrm{right}}.\n", @@ -1722,9 +1561,7 @@ { "cell_type": "markdown", "id": "ea62e615", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here the MSE for a specific node is defined as" ] @@ -1732,9 +1569,7 @@ { "cell_type": "markdown", "id": "8a20f4b4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathrm{MSE}_{\\mathrm{node}}=\\frac{1}{m_\\mathrm{node}}\\sum_{i\\in \\mathrm{node}}(\\overline{y}_{\\mathrm{node}}-y_i)^2,\n", @@ -1744,9 +1579,7 @@ { "cell_type": "markdown", "id": "9be4c8a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with" ] @@ -1754,9 +1587,7 @@ { "cell_type": "markdown", "id": "c4f7f184", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\overline{y}_{\\mathrm{node}}=\\frac{1}{m_\\mathrm{node}}\\sum_{i\\in \\mathrm{node}}y_i,\n", @@ -1766,9 +1597,7 @@ { "cell_type": "markdown", "id": "95d81e34", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "the mean value of all observations in a specific node.\n", "\n", @@ -1779,9 +1608,7 @@ { "cell_type": "markdown", "id": "e00e4c74", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Computing the Gini index\n", "\n", @@ -1823,9 +1650,7 @@ { "cell_type": "markdown", "id": "6f87ddeb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple Python Code to read in Data and perform Classification" ] @@ -1834,10 +1659,7 @@ "cell_type": "code", "execution_count": 13, "id": "9a2a4a21", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Common imports\n", @@ -1912,9 +1734,7 @@ { "cell_type": "markdown", "id": "167cae6a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Computing the Gini Factor\n", "\n", @@ -1930,10 +1750,7 @@ "cell_type": "code", "execution_count": 14, "id": "fd4d3d1d", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Split a dataset based on an attribute and an attribute value\n", @@ -2001,9 +1818,7 @@ { "cell_type": "markdown", "id": "e7637723", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Entropy and the ID3 algorithm\n", "\n", @@ -2040,9 +1855,7 @@ { "cell_type": "markdown", "id": "5fe05c6a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Cancer Data again now with Decision Trees and other Methods" ] @@ -2051,10 +1864,7 @@ "cell_type": "code", "execution_count": 15, "id": "d07cc03c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -2103,9 +1913,7 @@ { "cell_type": "markdown", "id": "ac909d39", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Another example, the moons again" ] @@ -2114,10 +1922,7 @@ "cell_type": "code", "execution_count": 16, "id": "4ec06694", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from __future__ import division, print_function, unicode_literals\n", @@ -2189,9 +1994,7 @@ { "cell_type": "markdown", "id": "b6e84ca9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Playing around with regions" ] @@ -2200,10 +2003,7 @@ "cell_type": "code", "execution_count": 17, "id": "3b8265d3", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "np.random.seed(6)\n", @@ -2231,9 +2031,7 @@ { "cell_type": "markdown", "id": "4ac4391c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Regression trees" ] @@ -2242,10 +2040,7 @@ "cell_type": "code", "execution_count": 18, "id": "0399291c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Quadratic training set + noise\n", @@ -2260,10 +2055,7 @@ "cell_type": "code", "execution_count": 19, "id": "6273a4ca", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.tree import DecisionTreeRegressor\n", @@ -2275,9 +2067,7 @@ { "cell_type": "markdown", "id": "500c38f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final regressor code" ] @@ -2286,10 +2076,7 @@ "cell_type": "code", "execution_count": 20, "id": "cff275f0", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.tree import DecisionTreeRegressor\n", @@ -2336,10 +2123,7 @@ "cell_type": "code", "execution_count": 21, "id": "d00f1eec", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "tree_reg1 = DecisionTreeRegressor(random_state=42)\n", @@ -2375,9 +2159,7 @@ { "cell_type": "markdown", "id": "efe7595b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Pros and cons of trees, pros\n", "\n", @@ -2399,9 +2181,7 @@ { "cell_type": "markdown", "id": "bd019bce", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Disadvantages\n", "\n", @@ -2427,9 +2207,7 @@ { "cell_type": "markdown", "id": "26879a96", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods\n", "\n", @@ -2458,9 +2236,7 @@ { "cell_type": "markdown", "id": "c4186bbd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## An Overview of Ensemble Methods\n", "\n", @@ -2474,9 +2250,7 @@ { "cell_type": "markdown", "id": "adf5117b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bagging\n", "\n", @@ -2496,9 +2270,7 @@ { "cell_type": "markdown", "id": "7e7b7e57", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More bagging\n", "\n", @@ -2528,9 +2300,7 @@ { "cell_type": "markdown", "id": "49a7dff2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple Voting Example, head or tail" ] @@ -2539,10 +2309,7 @@ "cell_type": "code", "execution_count": 22, "id": "0a443e4d", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "heads_proba = 0.51\n", @@ -2563,9 +2330,7 @@ { "cell_type": "markdown", "id": "a8ea734f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using the Voting Classifier" ] @@ -2574,10 +2339,7 @@ "cell_type": "code", "execution_count": 23, "id": "b292193c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", @@ -2628,9 +2390,7 @@ { "cell_type": "markdown", "id": "dd8452bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Please, not the moons again! Voting and Bagging" ] @@ -2639,10 +2399,7 @@ "cell_type": "code", "execution_count": 24, "id": "d8da9ef9", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", @@ -2669,10 +2426,7 @@ "cell_type": "code", "execution_count": 25, "id": "c6c0b366", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score\n", @@ -2687,10 +2441,7 @@ "cell_type": "code", "execution_count": 26, "id": "a7f6b0f8", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "log_clf = LogisticRegression(random_state=42)\n", @@ -2707,10 +2458,7 @@ "cell_type": "code", "execution_count": 27, "id": "b417ccb8", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score\n", @@ -2724,9 +2472,7 @@ { "cell_type": "markdown", "id": "3c34873d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bagging Examples" ] @@ -2735,10 +2481,7 @@ "cell_type": "code", "execution_count": 28, "id": "7d9235a9", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.ensemble import BaggingClassifier\n", @@ -2755,10 +2498,7 @@ "cell_type": "code", "execution_count": 29, "id": "fa17aa74", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score\n", @@ -2769,10 +2509,7 @@ "cell_type": "code", "execution_count": 30, "id": "8f65f7db", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "tree_clf = DecisionTreeClassifier(random_state=42)\n", @@ -2785,10 +2522,7 @@ "cell_type": "code", "execution_count": 31, "id": "dd292ec6", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from matplotlib.colors import ListedColormap\n", @@ -2823,9 +2557,7 @@ { "cell_type": "markdown", "id": "b8aa3665", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Making your own Bootstrap: Changing the Level of the Decision Tree\n", "\n", @@ -2837,10 +2569,7 @@ "cell_type": "code", "execution_count": 32, "id": "d15e0b7c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -2904,7 +2633,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" + } + }, "nbformat": 4, "nbformat_minor": 5 }