lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

typos in decision trees

Browse Source
This commit is contained in:
mhjensen
2019-12-05 22:01:09 +01:00
parent 50dfe24624
commit ac26951bd1
14 changed files with 72 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DecisionTrees/html/._DecisionTrees-bs000.html
+1 -1
View File
@@ -297,7 +297,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
<center><h4>Dec 5, 2019</h4></center> <!-- date -->
<br>
<p>
doc/pub/DecisionTrees/html/._DecisionTrees-bs046.html
+3 -3
View File
@@ -305,7 +305,7 @@ $$
<p>
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and
\( \gamma_1 \) where determined by the Logistic Regression fitting
\( \gamma_1 \) were determined by the Logistic Regression fitting
algorithm.
<p>
@@ -317,8 +317,8 @@ $$
<p>
In this case the function \( f(x) \) was replaced by the design matrix
\( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \),
that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we could
simply invert a matrix and obtained the parameters \( \beta \) by
that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can
simply invert a matrix and obtain the parameters \( \beta \) by
$$
\boldsymbol{\beta}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
doc/pub/DecisionTrees/html/._DecisionTrees-bs048.html
+2 -2
View File
@@ -287,10 +287,10 @@ To better understand what happens, let us develop the steps for the iterative fi
For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
<p>
This means that for every iteration, we need to optimize
This means that for every iteration \( m \), we need to optimize
$$
(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
$$
<p>
doc/pub/DecisionTrees/html/._DecisionTrees-bs049.html
+2 -2
View File
@@ -282,7 +282,7 @@ MathJax.Hub.Config({
<p>
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values
observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values
\( \{-1,1\} \).
<p>
@@ -295,7 +295,7 @@ $$
<p>
The iterative procedure starts with defining a weak classifier whose
error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
procedure in boosting is to sequentially apply a weak
classification algorithm to repeatedly modified versions of the data
producing a sequence of weak classifiers \( G_m(x) \).
doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html
+3 -3
View File
@@ -285,7 +285,7 @@ With the above definitions we are now ready to set up the algorithm for AdaBoost
The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
<ol>
<li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is to see then that \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is easy to see that we must have \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<li> We rewrite the misclassification error as</li>
</ol>
@@ -298,7 +298,7 @@ $$
<li> Then we start looping over all attempts at classifying, namely we start an iterative process for \( m=1:M \), where \( M \) is the final number of classifications. Our given classifier could for example be a plain decision tree.
<ol type="a"></li>
<li> Fit then a given classifier to the training using the weights \( w_i \).</li>
<li> Fit then a given classifier to the training set using the weights \( w_i \).</li>
<li> Compute then \( \mathrm{err} \) and figure out which events are classified properly and which are classified wrongly.</li>
<li> Define a quantity \( \alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m} \)</li>
<li> Set the new weights to \( w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)} \).</li>
@@ -308,7 +308,7 @@ $$
</ol>
For the iterations with \( m \le 2 \) the weights are modified
individually at each steps. The obersvations which were misclassified
individually at each steps. The observations which were misclassified
at iteration \( m-1 \) have a weight which is larger than those which were
classified properly. As this proceeds, the observations which were
difficult to classifiy correctly are given a larger influence. Each
doc/pub/DecisionTrees/html/._DecisionTrees-bs054.html
+1 -1
View File
@@ -281,7 +281,7 @@ MathJax.Hub.Config({
<h2 id="___sec53" class="anchor">AdaBoost Examples </h2>
<p>
Using <b>Scikit-Learn</b> it is easy to appply the adaptive boosting algorithm, as done here.
Using <b>Scikit-Learn</b> it is easy to apply the adaptive boosting algorithm, as done here.
<p>
doc/pub/DecisionTrees/html/DecisionTrees-bs.html
+1 -1
View File
@@ -297,7 +297,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
<center><h4>Dec 5, 2019</h4></center> <!-- date -->
<br>
<p>
doc/pub/DecisionTrees/html/DecisionTrees-reveal.html
+12 -12
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
<center><h4>Dec 5, 2019</h4></center> <!-- date -->
<br>
<p>
@@ -2072,7 +2072,7 @@ $$
<p>
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and
\( \gamma_1 \) where determined by the Logistic Regression fitting
\( \gamma_1 \) were determined by the Logistic Regression fitting
algorithm.
<p>
@@ -2086,8 +2086,8 @@ $$
<p>
In this case the function \( f(x) \) was replaced by the design matrix
\( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \),
that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we could
simply invert a matrix and obtained the parameters \( \beta \) by
that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can
simply invert a matrix and obtain the parameters \( \beta \) by
<p>&nbsp;<br>
$$
@@ -2136,11 +2136,11 @@ To better understand what happens, let us develop the steps for the iterative fi
For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
<p>
This means that for every iteration, we need to optimize
This means that for every iteration \( m \), we need to optimize
<p>&nbsp;<br>
$$
(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
$$
<p>&nbsp;<br>
@@ -2189,7 +2189,7 @@ The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma
<p>
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values
observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values
\( \{-1,1\} \).
<p>
@@ -2204,7 +2204,7 @@ $$
<p>
The iterative procedure starts with defining a weak classifier whose
error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
procedure in boosting is to sequentially apply a weak
classification algorithm to repeatedly modified versions of the data
producing a sequence of weak classifiers \( G_m(x) \).
@@ -2349,7 +2349,7 @@ With the above definitions we are now ready to set up the algorithm for AdaBoost
The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
<ol>
<p><li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is to see then that \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<p><li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is easy to see that we must have \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<p><li> We rewrite the misclassification error as</li>
</ol>
<p>&nbsp;<br>
@@ -2363,7 +2363,7 @@ $$
<p><li> Then we start looping over all attempts at classifying, namely we start an iterative process for \( m=1:M \), where \( M \) is the final number of classifications. Our given classifier could for example be a plain decision tree.
<ol type="a"></li>
<p><li> Fit then a given classifier to the training using the weights \( w_i \).</li>
<p><li> Fit then a given classifier to the training set using the weights \( w_i \).</li>
<p><li> Compute then \( \mathrm{err} \) and figure out which events are classified properly and which are classified wrongly.</li>
<p><li> Define a quantity \( \alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m} \)</li>
<p><li> Set the new weights to \( w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)} \).</li>
@@ -2373,7 +2373,7 @@ $$
<p>
For the iterations with \( m \le 2 \) the weights are modified
individually at each steps. The obersvations which were misclassified
individually at each steps. The observations which were misclassified
at iteration \( m-1 \) have a weight which is larger than those which were
classified properly. As this proceeds, the observations which were
difficult to classifiy correctly are given a larger influence. Each
@@ -2386,7 +2386,7 @@ observations that are missed in the previous iterations.
<h2 id="___sec53">AdaBoost Examples </h2>
<p>
Using <b>Scikit-Learn</b> it is easy to appply the adaptive boosting algorithm, as done here.
Using <b>Scikit-Learn</b> it is easy to apply the adaptive boosting algorithm, as done here.
<p>
doc/pub/DecisionTrees/html/DecisionTrees-solarized.html
+12 -12
View File
@@ -223,7 +223,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
<center><h4>Dec 5, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2080,7 +2080,7 @@ $$
<p>
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and
\( \gamma_1 \) where determined by the Logistic Regression fitting
\( \gamma_1 \) were determined by the Logistic Regression fitting
algorithm.
<p>
@@ -2092,8 +2092,8 @@ $$
<p>
In this case the function \( f(x) \) was replaced by the design matrix
\( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \),
that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we could
simply invert a matrix and obtained the parameters \( \beta \) by
that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can
simply invert a matrix and obtain the parameters \( \beta \) by
$$
\boldsymbol{\beta}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
@@ -2139,10 +2139,10 @@ To better understand what happens, let us develop the steps for the iterative fi
For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
<p>
This means that for every iteration, we need to optimize
This means that for every iteration \( m \), we need to optimize
$$
(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
$$
<p>
@@ -2182,7 +2182,7 @@ The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma
<p>
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values
observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values
\( \{-1,1\} \).
<p>
@@ -2195,7 +2195,7 @@ $$
<p>
The iterative procedure starts with defining a weak classifier whose
error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
procedure in boosting is to sequentially apply a weak
classification algorithm to repeatedly modified versions of the data
producing a sequence of weak classifiers \( G_m(x) \).
@@ -2313,7 +2313,7 @@ With the above definitions we are now ready to set up the algorithm for AdaBoost
The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
<ol>
<li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is to see then that \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is easy to see that we must have \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<li> We rewrite the misclassification error as</li>
</ol>
@@ -2326,7 +2326,7 @@ $$
<li> Then we start looping over all attempts at classifying, namely we start an iterative process for \( m=1:M \), where \( M \) is the final number of classifications. Our given classifier could for example be a plain decision tree.
<ol type="a"></li>
<li> Fit then a given classifier to the training using the weights \( w_i \).</li>
<li> Fit then a given classifier to the training set using the weights \( w_i \).</li>
<li> Compute then \( \mathrm{err} \) and figure out which events are classified properly and which are classified wrongly.</li>
<li> Define a quantity \( \alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m} \)</li>
<li> Set the new weights to \( w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)} \).</li>
@@ -2336,7 +2336,7 @@ $$
</ol>
For the iterations with \( m \le 2 \) the weights are modified
individually at each steps. The obersvations which were misclassified
individually at each steps. The observations which were misclassified
at iteration \( m-1 \) have a weight which is larger than those which were
classified properly. As this proceeds, the observations which were
difficult to classifiy correctly are given a larger influence. Each
@@ -2349,7 +2349,7 @@ observations that are missed in the previous iterations.
<h2 id="___sec53">AdaBoost Examples </h2>
<p>
Using <b>Scikit-Learn</b> it is easy to appply the adaptive boosting algorithm, as done here.
Using <b>Scikit-Learn</b> it is easy to apply the adaptive boosting algorithm, as done here.
<p>
doc/pub/DecisionTrees/html/DecisionTrees.html
+12 -12
View File
@@ -228,7 +228,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 24, 2019</h4></center> <!-- date -->
<center><h4>Dec 5, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2085,7 +2085,7 @@ $$
<p>
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and
\( \gamma_1 \) where determined by the Logistic Regression fitting
\( \gamma_1 \) were determined by the Logistic Regression fitting
algorithm.
<p>
@@ -2097,8 +2097,8 @@ $$
<p>
In this case the function \( f(x) \) was replaced by the design matrix
\( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \),
that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we could
simply invert a matrix and obtained the parameters \( \beta \) by
that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can
simply invert a matrix and obtain the parameters \( \beta \) by
$$
\boldsymbol{\beta}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
@@ -2144,10 +2144,10 @@ To better understand what happens, let us develop the steps for the iterative fi
For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
<p>
This means that for every iteration, we need to optimize
This means that for every iteration \( m \), we need to optimize
$$
(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
$$
<p>
@@ -2187,7 +2187,7 @@ The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma
<p>
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of
observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values
observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values
\( \{-1,1\} \).
<p>
@@ -2200,7 +2200,7 @@ $$
<p>
The iterative procedure starts with defining a weak classifier whose
error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
procedure in boosting is to sequentially apply a weak
classification algorithm to repeatedly modified versions of the data
producing a sequence of weak classifiers \( G_m(x) \).
@@ -2318,7 +2318,7 @@ With the above definitions we are now ready to set up the algorithm for AdaBoost
The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
<ol>
<li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is to see then that \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<li> We start by initializing all weights to \( w_i = 1/n \), with \( i=0,1,2,\dots n-1 \). It is easy to see that we must have \( \sum_{i=0}^{n-1}w_i = 1 \).</li>
<li> We rewrite the misclassification error as</li>
</ol>
@@ -2331,7 +2331,7 @@ $$
<li> Then we start looping over all attempts at classifying, namely we start an iterative process for \( m=1:M \), where \( M \) is the final number of classifications. Our given classifier could for example be a plain decision tree.
<ol type="a"></li>
<li> Fit then a given classifier to the training using the weights \( w_i \).</li>
<li> Fit then a given classifier to the training set using the weights \( w_i \).</li>
<li> Compute then \( \mathrm{err} \) and figure out which events are classified properly and which are classified wrongly.</li>
<li> Define a quantity \( \alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m} \)</li>
<li> Set the new weights to \( w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)} \).</li>
@@ -2341,7 +2341,7 @@ $$
</ol>
For the iterations with \( m \le 2 \) the weights are modified
individually at each steps. The obersvations which were misclassified
individually at each steps. The observations which were misclassified
at iteration \( m-1 \) have a weight which is larger than those which were
classified properly. As this proceeds, the observations which were
difficult to classifiy correctly are given a larger influence. Each
@@ -2354,7 +2354,7 @@ observations that are missed in the previous iterations.
<h2 id="___sec53">AdaBoost Examples </h2>
<p>
Using <b>Scikit-Learn</b> it is easy to appply the adaptive boosting algorithm, as done here.
Using <b>Scikit-Learn</b> it is easy to apply the adaptive boosting algorithm, as done here.
<p>
doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
+12 -12
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Nov 24, 2019**\n",
"Date: **Dec 5, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -2145,7 +2145,7 @@
"metadata": {},
"source": [
"where $t=\\gamma_0+\\gamma_1 x$ and the parameters $\\gamma_0$ and\n",
"$\\gamma_1$ where determined by the Logistic Regression fitting\n",
"$\\gamma_1$ were determined by the Logistic Regression fitting\n",
"algorithm.\n",
"\n",
"As another example, consider the cost function we defined for linear regression"
@@ -2166,8 +2166,8 @@
"source": [
"In this case the function $f(x)$ was replaced by the design matrix\n",
"$\\boldsymbol{X}$ and the unknown linear regression parameters $\\boldsymbol{\\beta}$,\n",
"that is $\\boldsymbol{f}=\\boldsymbol{X}\\boldsymbol{\\beta}$. In linear regression we could\n",
"simply invert a matrix and obtained the parameters $\\beta$ by"
"that is $\\boldsymbol{f}=\\boldsymbol{X}\\boldsymbol{\\beta}$. In linear regression we can \n",
"simply invert a matrix and obtain the parameters $\\beta$ by"
]
},
{
@@ -2214,7 +2214,7 @@
"\n",
"For simplicity we assume also that our functions $b(x;\\gamma)=1+\\gamma x$. \n",
"\n",
"This means that for every iteration, we need to optimize"
"This means that for every iteration $m$, we need to optimize"
]
},
{
@@ -2222,7 +2222,7 @@
"metadata": {},
"source": [
"$$\n",
"(\\beta_m,\\gamma_m) \\mathrm{argmin}_{\\beta,\\lambda}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2=\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(1+\\gamma x_i))^2.\n",
"(\\beta_m,\\gamma_m) = \\mathrm{argmin}_{\\beta,\\lambda}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2=\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(1+\\gamma x_i))^2.\n",
"$$"
]
},
@@ -2306,7 +2306,7 @@
"## Iterative Fitting, Classification and AdaBoost\n",
"\n",
"Let us consider a binary classification problem with two outcomes $y_i \\in \\{-1,1\\}$ and $i=0,1,2,\\dots,n-1$ as our set of\n",
"observations. We define a classification function $G(x)$ which produces a prediction taking any of the two values \n",
"observations. We define a classification function $G(x)$ which produces a prediction taking one or the other of the two values \n",
"$\\{-1,1\\}$.\n",
"\n",
"The error rate of the training sample is then"
@@ -2327,7 +2327,7 @@
"source": [
"The iterative procedure starts with defining a weak classifier whose\n",
"error rate is barely better than random guessing. The iterative\n",
"procedure in boosting is to sequentially apply a the weak\n",
"procedure in boosting is to sequentially apply a weak\n",
"classification algorithm to repeatedly modified versions of the data\n",
"producing a sequence of weak classifiers $G_m(x)$.\n",
"\n",
@@ -2565,7 +2565,7 @@
"\n",
"With the above definitions we are now ready to set up the algorithm for AdaBoost.\n",
"The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.\n",
"1. We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\\dots n-1$. It is to see then that $\\sum_{i=0}^{n-1}w_i = 1$.\n",
"1. We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\\dots n-1$. It is easy to see that we must have $\\sum_{i=0}^{n-1}w_i = 1$.\n",
"\n",
"2. We rewrite the misclassification error as"
]
@@ -2585,7 +2585,7 @@
"source": [
"1. Then we start looping over all attempts at classifying, namely we start an iterative process for $m=1:M$, where $M$ is the final number of classifications. Our given classifier could for example be a plain decision tree.\n",
"\n",
"a. Fit then a given classifier to the training using the weights $w_i$.\n",
"a. Fit then a given classifier to the training set using the weights $w_i$.\n",
"\n",
"b. Compute then $\\mathrm{err}$ and figure out which events are classified properly and which are classified wrongly.\n",
"\n",
@@ -2597,7 +2597,7 @@
"5. Compute the new classifier $G(x)= \\sum_{i=0}^{n-1}\\alpha_m I(y_i\\ne G(x_i)$.\n",
"\n",
"For the iterations with $m \\le 2$ the weights are modified\n",
"individually at each steps. The obersvations which were misclassified\n",
"individually at each steps. The observations which were misclassified\n",
"at iteration $m-1$ have a weight which is larger than those which were\n",
"classified properly. As this proceeds, the observations which were\n",
"difficult to classifiy correctly are given a larger influence. Each\n",
@@ -2608,7 +2608,7 @@
"\n",
"## AdaBoost Examples\n",
"\n",
"Using **Scikit-Learn** it is easy to appply the adaptive boosting algorithm, as done here."
"Using **Scikit-Learn** it is easy to apply the adaptive boosting algorithm, as done here."
]
},
{
doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf
BIN
View File
Binary file not shown.
doc/src/DecisionTrees/DecisionTrees.do.txt
+11 -11
View File
@@ -1709,7 +1709,7 @@ $b(x;\gamma_m)$ into the Sigmoid function
!et
where $t=\gamma_0+\gamma_1 x$ and the parameters $\gamma_0$ and
$\gamma_1$ where determined by the Logistic Regression fitting
$\gamma_1$ were determined by the Logistic Regression fitting
algorithm.
As another example, consider the cost function we defined for linear regression
@@ -1721,8 +1721,8 @@ C(\bm{y},\bm{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
In this case the function $f(x)$ was replaced by the design matrix
$\bm{X}$ and the unknown linear regression parameters $\bm{\beta}$,
that is $\bm{f}=\bm{X}\bm{\beta}$. In linear regression we could
simply invert a matrix and obtained the parameters $\beta$ by
that is $\bm{f}=\bm{X}\bm{\beta}$. In linear regression we can
simply invert a matrix and obtain the parameters $\beta$ by
!bt
\[
@@ -1757,11 +1757,11 @@ To better understand what happens, let us develop the steps for the iterative fi
For simplicity we assume also that our functions $b(x;\gamma)=1+\gamma x$.
This means that for every iteration, we need to optimize
This means that for every iteration $m$, we need to optimize
!bt
\[
(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
\]
!et
@@ -1803,7 +1803,7 @@ $f_1(x) = \beta_1(1+\gamma_1x)$. Doing this $M$ times results in our final estim
===== Iterative Fitting, Classification and AdaBoost =====
Let us consider a binary classification problem with two outcomes $y_i \in \{-1,1\}$ and $i=0,1,2,\dots,n-1$ as our set of
observations. We define a classification function $G(x)$ which produces a prediction taking any of the two values
observations. We define a classification function $G(x)$ which produces a prediction taking one or the other of the two values
$\{-1,1\}$.
The error rate of the training sample is then
@@ -1816,7 +1816,7 @@ The error rate of the training sample is then
The iterative procedure starts with defining a weak classifier whose
error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a the weak
procedure in boosting is to sequentially apply a weak
classification algorithm to repeatedly modified versions of the data
producing a sequence of weak classifiers $G_m(x)$.
@@ -1934,7 +1934,7 @@ where the function $I()$ is one if we misclassify and zero if we classify correc
With the above definitions we are now ready to set up the algorithm for AdaBoost.
The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
o We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\dots n-1$. It is to see then that $\sum_{i=0}^{n-1}w_i = 1$.
o We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\dots n-1$. It is easy to see that we must have $\sum_{i=0}^{n-1}w_i = 1$.
o We rewrite the misclassification error as
!bt
\[
@@ -1942,14 +1942,14 @@ o We rewrite the misclassification error as
\]
!et
o Then we start looping over all attempts at classifying, namely we start an iterative process for $m=1:M$, where $M$ is the final number of classifications. Our given classifier could for example be a plain decision tree.
o Fit then a given classifier to the training using the weights $w_i$.
o Fit then a given classifier to the training set using the weights $w_i$.
o Compute then $\mathrm{err}$ and figure out which events are classified properly and which are classified wrongly.
o Define a quantity $\alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m}$
o Set the new weights to $w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)}$.
o Compute the new classifier $G(x)= \sum_{i=0}^{n-1}\alpha_m I(y_i\ne G(x_i)$.
For the iterations with $m \le 2$ the weights are modified
individually at each steps. The obersvations which were misclassified
individually at each steps. The observations which were misclassified
at iteration $m-1$ have a weight which is larger than those which were
classified properly. As this proceeds, the observations which were
difficult to classifiy correctly are given a larger influence. Each
@@ -1961,7 +1961,7 @@ observations that are missed in the previous iterations.
!split
===== AdaBoost Examples =====
Using _Scikit-Learn_ it is easy to appply the adaptive boosting algorithm, as done here.
Using _Scikit-Learn_ it is easy to apply the adaptive boosting algorithm, as done here.
!bc pycod
from sklearn.ensemble import AdaBoostClassifier