Forgot to update CART also
This commit is contained in:
mhjensen
@@ -297,7 +297,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Nov 13, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
|
||||
@@ -288,7 +288,10 @@ Two algorithms stand out in the set up of decision trees:
|
||||
<li> The ID3 algorithm based on the computation of the information gain for classification</li>
|
||||
</ol>
|
||||
|
||||
We discuss both algorithms with applications here. The popular library <b>Scikit-Learn</b> uses the CART algorithm. For classification problems you can use either the <b>gini</b> index or the <b>entropy</b> to split a tree in two branches.
|
||||
We discuss both algorithms with applications here. The popular library
|
||||
<b>Scikit-Learn</b> uses the CART algorithm. For classification problems
|
||||
you can use either the <b>gini</b> index or the <b>entropy</b> to split a tree
|
||||
in two branches.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
|
||||
@@ -280,6 +280,28 @@ MathJax.Hub.Config({
|
||||
|
||||
<h2 id="___sec17" class="anchor">The CART algorithm for Classification </h2>
|
||||
|
||||
<p>
|
||||
For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
|
||||
This could be for example a threshold set by a number below a certain circumference of a malign tumor.
|
||||
|
||||
<p>
|
||||
How do we find these two quantities?
|
||||
We search for the pair \( (k,t_k) \) that produces the purest subset using for example the <b>gini</b> factor \( G \).
|
||||
The cost function it tries to minimize is then
|
||||
$$
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
|
||||
$$
|
||||
|
||||
where \( G_{\mathrm{left\left}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \)
|
||||
is the number of instances in the left/right subset
|
||||
|
||||
<p>
|
||||
Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
|
||||
and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
|
||||
\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other
|
||||
hyperparameters control additional stopping conditions such as the \( min\_samples\_split \),
|
||||
\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
|
||||
@@ -280,6 +280,29 @@ MathJax.Hub.Config({
|
||||
|
||||
<h2 id="___sec18" class="anchor">The CART algorithm for Regression </h2>
|
||||
|
||||
<p>
|
||||
The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
|
||||
training set in a way that minimizes say the <b>gini</b> or <b>entropy</b> impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now
|
||||
$$
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}.
|
||||
$$
|
||||
|
||||
Here the MSE for a specific node is defined as
|
||||
$$
|
||||
\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
|
||||
$$
|
||||
|
||||
with
|
||||
$$
|
||||
\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
|
||||
$$
|
||||
|
||||
the mean value of all observations in a specific node.
|
||||
|
||||
<p>
|
||||
Without any regularization, the regression task for decision trees,
|
||||
just like for classification tasks, is prone to overfitting.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
|
||||
@@ -297,7 +297,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Nov 13, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
|
||||
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p> <br>
|
||||
<center><h4>Nov 13, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -708,17 +708,73 @@ Two algorithms stand out in the set up of decision trees:
|
||||
</ol>
|
||||
<p>
|
||||
|
||||
We discuss both algorithms with applications here. The popular library <b>Scikit-Learn</b> uses the CART algorithm. For classification problems you can use either the <b>gini</b> index or the <b>entropy</b> to split a tree in two branches.
|
||||
We discuss both algorithms with applications here. The popular library
|
||||
<b>Scikit-Learn</b> uses the CART algorithm. For classification problems
|
||||
you can use either the <b>gini</b> index or the <b>entropy</b> to split a tree
|
||||
in two branches.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec17">The CART algorithm for Classification </h2>
|
||||
|
||||
<p>
|
||||
For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
|
||||
This could be for example a threshold set by a number below a certain circumference of a malign tumor.
|
||||
|
||||
<p>
|
||||
How do we find these two quantities?
|
||||
We search for the pair \( (k,t_k) \) that produces the purest subset using for example the <b>gini</b> factor \( G \).
|
||||
The cost function it tries to minimize is then
|
||||
<p> <br>
|
||||
$$
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
where \( G_{\mathrm{left\left}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \)
|
||||
is the number of instances in the left/right subset
|
||||
|
||||
<p>
|
||||
Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
|
||||
and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
|
||||
\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other
|
||||
hyperparameters control additional stopping conditions such as the \( min\_samples\_split \),
|
||||
\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \).
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec18">The CART algorithm for Regression </h2>
|
||||
|
||||
<p>
|
||||
The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
|
||||
training set in a way that minimizes say the <b>gini</b> or <b>entropy</b> impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now
|
||||
<p> <br>
|
||||
$$
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
Here the MSE for a specific node is defined as
|
||||
<p> <br>
|
||||
$$
|
||||
\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
with
|
||||
<p> <br>
|
||||
$$
|
||||
\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
the mean value of all observations in a specific node.
|
||||
|
||||
<p>
|
||||
Without any regularization, the regression task for decision trees,
|
||||
just like for classification tasks, is prone to overfitting.
|
||||
</section>
|
||||
|
||||
|
||||
|
||||
@@ -223,7 +223,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Nov 13, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -755,18 +755,66 @@ Two algorithms stand out in the set up of decision trees:
|
||||
<li> The ID3 algorithm based on the computation of the information gain for classification</li>
|
||||
</ol>
|
||||
|
||||
We discuss both algorithms with applications here. The popular library <b>Scikit-Learn</b> uses the CART algorithm. For classification problems you can use either the <b>gini</b> index or the <b>entropy</b> to split a tree in two branches.
|
||||
We discuss both algorithms with applications here. The popular library
|
||||
<b>Scikit-Learn</b> uses the CART algorithm. For classification problems
|
||||
you can use either the <b>gini</b> index or the <b>entropy</b> to split a tree
|
||||
in two branches.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec17">The CART algorithm for Classification </h2>
|
||||
|
||||
<p>
|
||||
For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
|
||||
This could be for example a threshold set by a number below a certain circumference of a malign tumor.
|
||||
|
||||
<p>
|
||||
How do we find these two quantities?
|
||||
We search for the pair \( (k,t_k) \) that produces the purest subset using for example the <b>gini</b> factor \( G \).
|
||||
The cost function it tries to minimize is then
|
||||
$$
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
|
||||
$$
|
||||
|
||||
where \( G_{\mathrm{left\left}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \)
|
||||
is the number of instances in the left/right subset
|
||||
|
||||
<p>
|
||||
Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
|
||||
and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
|
||||
\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other
|
||||
hyperparameters control additional stopping conditions such as the \( min\_samples\_split \),
|
||||
\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">The CART algorithm for Regression </h2>
|
||||
|
||||
<p>
|
||||
The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
|
||||
training set in a way that minimizes say the <b>gini</b> or <b>entropy</b> impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now
|
||||
$$
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}.
|
||||
$$
|
||||
|
||||
Here the MSE for a specific node is defined as
|
||||
$$
|
||||
\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
|
||||
$$
|
||||
|
||||
with
|
||||
$$
|
||||
\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
|
||||
$$
|
||||
|
||||
the mean value of all observations in a specific node.
|
||||
|
||||
<p>
|
||||
Without any regularization, the regression task for decision trees,
|
||||
just like for classification tasks, is prone to overfitting.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
|
||||
@@ -228,7 +228,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Nov 13, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -760,18 +760,66 @@ Two algorithms stand out in the set up of decision trees:
|
||||
<li> The ID3 algorithm based on the computation of the information gain for classification</li>
|
||||
</ol>
|
||||
|
||||
We discuss both algorithms with applications here. The popular library <b>Scikit-Learn</b> uses the CART algorithm. For classification problems you can use either the <b>gini</b> index or the <b>entropy</b> to split a tree in two branches.
|
||||
We discuss both algorithms with applications here. The popular library
|
||||
<b>Scikit-Learn</b> uses the CART algorithm. For classification problems
|
||||
you can use either the <b>gini</b> index or the <b>entropy</b> to split a tree
|
||||
in two branches.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec17">The CART algorithm for Classification </h2>
|
||||
|
||||
<p>
|
||||
For classification, the CART algorithm splits the data set in two subsets using a single feature \( k \) and a threshold \( t_k \).
|
||||
This could be for example a threshold set by a number below a certain circumference of a malign tumor.
|
||||
|
||||
<p>
|
||||
How do we find these two quantities?
|
||||
We search for the pair \( (k,t_k) \) that produces the purest subset using for example the <b>gini</b> factor \( G \).
|
||||
The cost function it tries to minimize is then
|
||||
$$
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
|
||||
$$
|
||||
|
||||
where \( G_{\mathrm{left\left}} \) measures the impurity of the left/right subset and \( m_{\mathrm{left/right}} \)
|
||||
is the number of instances in the left/right subset
|
||||
|
||||
<p>
|
||||
Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
|
||||
and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
|
||||
\( max\_depth \) hyperparameter), or if it cannot find a split that will reduce impurity. A few other
|
||||
hyperparameters control additional stopping conditions such as the \( min\_samples\_split \),
|
||||
\( min\_samples\_leaf \), \( min\_weight\_fraction\_leaf \), and \( max\_leaf\_nodes \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">The CART algorithm for Regression </h2>
|
||||
|
||||
<p>
|
||||
The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
|
||||
training set in a way that minimizes say the <b>gini</b> or <b>entropy</b> impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now
|
||||
$$
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}.
|
||||
$$
|
||||
|
||||
Here the MSE for a specific node is defined as
|
||||
$$
|
||||
\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
|
||||
$$
|
||||
|
||||
with
|
||||
$$
|
||||
\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
|
||||
$$
|
||||
|
||||
the mean value of all observations in a specific node.
|
||||
|
||||
<p>
|
||||
Without any regularization, the regression task for decision trees,
|
||||
just like for classification tasks, is prone to overfitting.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
"<!-- Author: --> \n",
|
||||
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
||||
"\n",
|
||||
"Date: **Nov 13, 2019**\n",
|
||||
"Date: **Nov 24, 2019**\n",
|
||||
"\n",
|
||||
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
||||
"\n",
|
||||
@@ -609,13 +609,98 @@
|
||||
"\n",
|
||||
"2. The ID3 algorithm based on the computation of the information gain for classification\n",
|
||||
"\n",
|
||||
"We discuss both algorithms with applications here. The popular library **Scikit-Learn** uses the CART algorithm. For classification problems you can use either the **gini** index or the **entropy** to split a tree in two branches.\n",
|
||||
"We discuss both algorithms with applications here. The popular library\n",
|
||||
"**Scikit-Learn** uses the CART algorithm. For classification problems\n",
|
||||
"you can use either the **gini** index or the **entropy** to split a tree\n",
|
||||
"in two branches.\n",
|
||||
"\n",
|
||||
"## The CART algorithm for Classification\n",
|
||||
"\n",
|
||||
"For classification, the CART algorithm splits the data set in two subsets using a single feature $k$ and a threshold $t_k$.\n",
|
||||
"This could be for example a threshold set by a number below a certain circumference of a malign tumor.\n",
|
||||
"\n",
|
||||
"How do we find these two quantities?\n",
|
||||
"We search for the pair $(k,t_k)$ that produces the purest subset using for example the **gini** factor $G$.\n",
|
||||
"The cost function it tries to minimize is then"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"C(k,t_k) = \\frac{m_{\\mathrm{left}}}{m}G_{\\mathrm{left}}+ \\frac{m_{\\mathrm{right}}}{m}G_{\\mathrm{right}},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"where $G_{\\mathrm{left\\left}}$ measures the impurity of the left/right subset and $m_{\\mathrm{left/right}}$\n",
|
||||
" is the number of instances in the left/right subset\n",
|
||||
"\n",
|
||||
"Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets\n",
|
||||
"and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the\n",
|
||||
"$max\\_depth$ hyperparameter), or if it cannot find a split that will reduce impurity. A few other\n",
|
||||
"hyperparameters control additional stopping conditions such as the $min\\_samples\\_split$,\n",
|
||||
"$min\\_samples\\_leaf$, $min\\_weight\\_fraction\\_leaf$, and $max\\_leaf\\_nodes$.\n",
|
||||
"\n",
|
||||
"## The CART algorithm for Regression\n",
|
||||
"\n",
|
||||
"The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the\n",
|
||||
"training set in a way that minimizes say the **gini** or **entropy** impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"C(k,t_k) = \\frac{m_{\\mathrm{left}}}{m}\\mathrm{MSE}_{\\mathrm{left}}+ \\frac{m_{\\mathrm{right}}}{m}\\mathrm{MSE}_{\\mathrm{right}}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here the MSE for a specific node is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\mathrm{MSE}_{\\mathrm{node}}=\\frac{1}{m_\\mathrm{node}}\\sum_{i\\in \\mathrm{node}}(\\overline{y}_{\\mathrm{node}}-y_i)^2,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"with"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\overline{y}_{\\mathrm{node}}=\\frac{1}{m_\\mathrm{node}}\\sum_{i\\in \\mathrm{node}}y_i,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"the mean value of all observations in a specific node.\n",
|
||||
"\n",
|
||||
"Without any regularization, the regression task for decision trees, \n",
|
||||
"just like for classification tasks, is prone to overfitting.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Computing the Gini index\n",
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -471,15 +471,60 @@ Two algorithms stand out in the set up of decision trees:
|
||||
o The CART (Classification And Regression Tree) algorithm for both classification and regression
|
||||
o The ID3 algorithm based on the computation of the information gain for classification
|
||||
|
||||
We discuss both algorithms with applications here. The popular library _Scikit-Learn_ uses the CART algorithm. For classification problems you can use either the _gini_ index or the _entropy_ to split a tree in two branches.
|
||||
We discuss both algorithms with applications here. The popular library
|
||||
_Scikit-Learn_ uses the CART algorithm. For classification problems
|
||||
you can use either the _gini_ index or the _entropy_ to split a tree
|
||||
in two branches.
|
||||
|
||||
!split
|
||||
===== The CART algorithm for Classification =====
|
||||
|
||||
For classification, the CART algorithm splits the data set in two subsets using a single feature $k$ and a threshold $t_k$.
|
||||
This could be for example a threshold set by a number below a certain circumference of a malign tumor.
|
||||
|
||||
How do we find these two quantities?
|
||||
We search for the pair $(k,t_k)$ that produces the purest subset using for example the _gini_ factor $G$.
|
||||
The cost function it tries to minimize is then
|
||||
!bt
|
||||
\[
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}G_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}G_{\mathrm{right}},
|
||||
\]
|
||||
!et
|
||||
where $G_{\mathrm{left\left}}$ measures the impurity of the left/right subset and $m_{\mathrm{left/right}}$
|
||||
is the number of instances in the left/right subset
|
||||
|
||||
Once it has successfully split the training set in two, it splits the subsets using the same logic, then the subsubsets
|
||||
and so on, recursively. It stops recursing once it reaches the maximum depth (defined by the
|
||||
$max\_depth$ hyperparameter), or if it cannot find a split that will reduce impurity. A few other
|
||||
hyperparameters control additional stopping conditions such as the $min\_samples\_split$,
|
||||
$min\_samples\_leaf$, $min\_weight\_fraction\_leaf$, and $max\_leaf\_nodes$.
|
||||
|
||||
!split
|
||||
===== The CART algorithm for Regression =====
|
||||
|
||||
The CART algorithm for regression works is similar to the one for classification except that instead of trying to split the
|
||||
training set in a way that minimizes say the _gini_ or _entropy_ impurity, it now tries to split the training set in a way that minimizes our well-known mean-squared error (MSE). The cost function is now
|
||||
!bt
|
||||
\[
|
||||
C(k,t_k) = \frac{m_{\mathrm{left}}}{m}\mathrm{MSE}_{\mathrm{left}}+ \frac{m_{\mathrm{right}}}{m}\mathrm{MSE}_{\mathrm{right}}.
|
||||
\]
|
||||
!et
|
||||
Here the MSE for a specific node is defined as
|
||||
!bt
|
||||
\[
|
||||
\mathrm{MSE}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}(\overline{y}_{\mathrm{node}}-y_i)^2,
|
||||
\]
|
||||
!et
|
||||
with
|
||||
!bt
|
||||
\[
|
||||
\overline{y}_{\mathrm{node}}=\frac{1}{m_\mathrm{node}}\sum_{i\in \mathrm{node}}y_i,
|
||||
\]
|
||||
!et
|
||||
the mean value of all observations in a specific node.
|
||||
|
||||
Without any regularization, the regression task for decision trees,
|
||||
just like for classification tasks, is prone to overfitting.
|
||||
|
||||
|
||||
!split
|
||||
|
||||
Reference in New Issue
Block a user