-
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs017.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs017.html index 97a71b5e7..dcc2c563b 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs017.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs017.html @@ -288,7 +288,10 @@ Two algorithms stand out in the set up of decision trees:
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs018.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs018.html index f106735bf..b8afc2173 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs018.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs018.html @@ -280,6 +280,28 @@ MathJax.Hub.Config({
+For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). +This could be for example a threshold set by a number below a certain circumference of a malign tumor. + +
+How do we find these two quantities? +We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \). +The cost function it tries to minimize is then +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}}, +$$ + +where \( G_{\mathrm{left\left}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \) + is the number of instances in the left/right subset + +
+Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets +and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the +\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other +hyperparameters control additional stopping conditions such as the \( min\_samples\_split \), +\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \). +
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs019.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs019.html index 424959151..19dc10456 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs019.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs019.html @@ -280,6 +280,29 @@ MathJax.Hub.Config({
+The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the +training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. +$$ + +Here the MSE for a specific node is defined as +$$ +\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2, +$$ + +with +$$ +\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i, +$$ + +the mean value of all observations in a specific node. + +
+Without any regularization, the regression task for decision trees, +just like for classification tasks, is prone to overfitting. +
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-bs.html b/doc/pub/DecisionTrees/html/DecisionTrees-bs.html index 054a16087..d84cbeccc 100644 --- a/doc/pub/DecisionTrees/html/DecisionTrees-bs.html +++ b/doc/pub/DecisionTrees/html/DecisionTrees-bs.html @@ -297,7 +297,7 @@ MathJax.Hub.Config({
-
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html b/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html index c2b91f094..b65e88beb 100644 --- a/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html +++ b/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
@@ -708,17 +708,73 @@ Two algorithms stand out in the set up of decision trees:
-We discuss both algorithms with applications here. The popular library Scikit-Learn uses the CART algorithm. For classification problems you can use either the gini index or the entropy to split a tree in two branches.
+We discuss both algorithms with applications here. The popular library
+Scikit-Learn uses the CART algorithm. For classification problems
+you can use either the gini index or the entropy to split a tree
+in two branches.
+For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
+This could be for example a threshold set by a number below a certain circumference of a malign tumor.
+
+
+How do we find these two quantities?
+We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \).
+The cost function it tries to minimize is then
+
+Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
+and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
+\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other
+hyperparameters control additional stopping conditions such as the \( min\_samples\_split \),
+\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \).
+The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
+training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now
+
+Without any regularization, the regression task for decision trees,
+just like for classification tasks, is prone to overfitting.
The CART algorithm for Classification
+
+
+$$
+C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
+$$
+
+
+where \( G_{\mathrm{left\left}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \)
+ is the number of instances in the left/right subset
+
+The CART algorithm for Regression
+
+
+$$
+C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}.
+$$
+
+
+Here the MSE for a specific node is defined as
+
+$$
+\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
+$$
+
+
+with
+
+$$
+\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
+$$
+
+
+the mean value of all observations in a specific node.
+
+
-
@@ -755,18 +755,66 @@ Two algorithms stand out in the set up of decision trees:
+For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). +This could be for example a threshold set by a number below a certain circumference of a malign tumor. + +
+How do we find these two quantities? +We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \). +The cost function it tries to minimize is then +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}}, +$$ + +where \( G_{\mathrm{left\left}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \) + is the number of instances in the left/right subset + +
+Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets +and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the +\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other +hyperparameters control additional stopping conditions such as the \( min\_samples\_split \), +\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \). +
+The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the +training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. +$$ + +Here the MSE for a specific node is defined as +$$ +\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2, +$$ + +with +$$ +\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i, +$$ + +the mean value of all observations in a specific node. + +
+Without any regularization, the regression task for decision trees, +just like for classification tasks, is prone to overfitting. +
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees.html b/doc/pub/DecisionTrees/html/DecisionTrees.html
index 61f9eb3dd..9d22847ca 100644
--- a/doc/pub/DecisionTrees/html/DecisionTrees.html
+++ b/doc/pub/DecisionTrees/html/DecisionTrees.html
@@ -228,7 +228,7 @@ MathJax.Hub.Config({
-
@@ -760,18 +760,66 @@ Two algorithms stand out in the set up of decision trees:
+For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \). +This could be for example a threshold set by a number below a certain circumference of a malign tumor. + +
+How do we find these two quantities? +We search for the pair \( (k,t_k) \) that produces the purest subset using for example the gini factor \( G \). +The cost function it tries to minimize is then +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}}, +$$ + +where \( G_{\mathrm{left\left}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \) + is the number of instances in the left/right subset + +
+Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets +and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the +\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other +hyperparameters control additional stopping conditions such as the \( min\_samples\_split \), +\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \). +
+The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the +training set in a way that minimizes say the gini or entropy impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now +$$ +C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}. +$$ + +Here the MSE for a specific node is defined as +$$ +\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2, +$$ + +with +$$ +\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i, +$$ + +the mean value of all observations in a specific node. + +
+Without any regularization, the regression task for decision trees, +just like for classification tasks, is prone to overfitting. +
diff --git a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
index 5f72bcd04..aaf2eaca4 100644
--- a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
+++ b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Nov 13, 2019**\n",
+ "Date: **Nov 24, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -609,13 +609,98 @@
"\n",
"2. The ID3 algorithm based on the computation of the information gain for classification\n",
"\n",
- "We discuss both algorithms with applications here. The popular library **Scikit-Learn** uses the CART algorithm. For classification problems you can use either the **gini** index or the **entropy** to split a tree in two branches.\n",
+ "We discuss both algorithms with applications here. The popular library\n",
+ "**Scikit-Learn** uses the CART algorithm. For classification problems\n",
+ "you can use either the **gini** index or the **entropy** to split a tree\n",
+ "in two branches.\n",
"\n",
"## The CART algorithm for Classification\n",
"\n",
+ "For classification, the CART algorithm splits the data set in two subsets using a single feature $k$ and a threshold $t_k$.\n",
+ "This could be for example a threshold set by a number below a certain circumference of a malign tumor.\n",
+ "\n",
+ "How do we find these two quantities?\n",
+ "We search for the pair $(k,t_k)$ that produces the purest subset using for example the **gini** factor $G$.\n",
+ "The cost function it tries to minimize is then"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(k,t_k) = \\frac{m_{\\mathrm{left}}}{m}G_{\\mathrm{left}}+ \\frac{m_{\\mathrm{right}}}{m}G_{\\mathrm{right}},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "where $G_{\\mathrm{left\\left}}$ measures the impurity of the left/right subset and $m_{\\mathrm{left/right}}$\n",
+ " is the number of instances in the left/right subset\n",
+ "\n",
+ "Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets\n",
+ "and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the\n",
+ "$max\\_depth$ hyperparameter), or if it cannot find a split that will reduce impurity. A few other\n",
+ "hyperparameters control additional stopping conditions such as the $min\\_samples\\_split$,\n",
+ "$min\\_samples\\_leaf$, $min\\_weight\\_fraction\\_leaf$, and $max\\_leaf\\_nodes$.\n",
"\n",
"## The CART algorithm for Regression\n",
"\n",
+ "The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the\n",
+ "training set in a way that minimizes say the **gini** or **entropy** impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(k,t_k) = \\frac{m_{\\mathrm{left}}}{m}\\mathrm{MSE}_{\\mathrm{left}}+ \\frac{m_{\\mathrm{right}}}{m}\\mathrm{MSE}_{\\mathrm{right}}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here the MSE for a specific node is defined as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\mathrm{MSE}_{\\mathrm{node}}=\\frac{1}{m_\\mathrm{node}}\\sum_{i\\in \\mathrm{node}}(\\overline{y}_{\\mathrm{node}}-y_i)^2,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "with"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\overline{y}_{\\mathrm{node}}=\\frac{1}{m_\\mathrm{node}}\\sum_{i\\in \\mathrm{node}}y_i,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "the mean value of all observations in a specific node.\n",
+ "\n",
+ "Without any regularization, the regression task for decision trees, \n",
+ "just like for classification tasks, is prone to overfitting.\n",
"\n",
"\n",
"## Computing the Gini index\n",
diff --git a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
index 43c2508b6..c08f488f1 100644
Binary files a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz and b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz differ
diff --git a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
index 519b8268c..6d8201a6e 100644
Binary files a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf and b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf differ
diff --git a/doc/src/DecisionTrees/DecisionTrees.do.txt b/doc/src/DecisionTrees/DecisionTrees.do.txt
index 1b2dd2ae9..c6f31cbd8 100644
--- a/doc/src/DecisionTrees/DecisionTrees.do.txt
+++ b/doc/src/DecisionTrees/DecisionTrees.do.txt
@@ -471,15 +471,60 @@ Two algorithms stand out in the set up of decision trees:
o The CART (Classification And Regression Tree) algorithm for both classification and regression
o The ID3 algorithm based on the computation of the information gain for classification
-We discuss both algorithms with applications here. The popular library _Scikit-Learn_ uses the CART algorithm. For classification problems you can use either the _gini_ index or the _entropy_ to split a tree in two branches.
+We discuss both algorithms with applications here. The popular library
+_Scikit-Learn_ uses the CART algorithm. For classification problems
+you can use either the _gini_ index or the _entropy_ to split a tree
+in two branches.
!split
===== The CART algorithm for Classification =====
+For classification, the CART algorithm splits the data set in two subsets using a single feature $k$ and a threshold $t_k$.
+This could be for example a threshold set by a number below a certain circumference of a malign tumor.
+
+How do we find these two quantities?
+We search for the pair $(k,t_k)$ that produces the purest subset using for example the _gini_ factor $G$.
+The cost function it tries to minimize is then
+!bt
+\[
+C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
+\]
+!et
+where $G_{\mathrm{left\left}}$ measures the impurity of the left/right subset and $m_{\mathrm{left/right}}$
+ is the number of instances in the left/right subset
+
+Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
+and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
+$max\_depth$ hyperparameter), or if it cannot find a split that will reduce impurity. A few other
+hyperparameters control additional stopping conditions such as the $min\_samples\_split$,
+$min\_samples\_leaf$, $min\_weight\_fraction\_leaf$, and $max\_leaf\_nodes$.
!split
===== The CART algorithm for Regression =====
+The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
+training set in a way that minimizes say the _gini_ or _entropy_ impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now
+!bt
+\[
+C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}.
+\]
+!et
+Here the MSE for a specific node is defined as
+!bt
+\[
+\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
+\]
+!et
+with
+!bt
+\[
+\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
+\]
+!et
+the mean value of all observations in a specific node.
+
+Without any regularization, the regression task for decision trees,
+just like for classification tasks, is prone to overfitting.
!split