more boosting stuff

doc/src/DecisionTrees/DecisionTrees.do.txt

@@ -1710,7 +1710,7 @@ This means that for every iteration, we need to optmize
 
 !bt
 \[
-(\beta_m,\gamma_m)  \mathrm{armmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
+(\beta_m,\gamma_m)  \mathrm{argmin}_{\beta,\gambda}\hspace{0.2cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(\gamma_0+\gamma_1 x_i))^2.
 \[
 !et
 
@@ -1724,17 +1724,23 @@ Taking the derivatives  with respect to $\beta$ and $\gamma$ we obtain
 and
 !bt
 \[
-\frac{\partial {\cal C}}{\partial \gamma_0} = 
+\frac{\partial {\cal C}}{\partial \gamma_0} =-2\sum_{i}\beta(y_i-\beta(\gamma_0+\gamma_1 x_i))=0,
 \]
 !et
 and
 !bt
 \[
-\frac{\partial {\cal C}}{\partial \gamma_1} = 
+\frac{\partial {\cal C}}{\partial \gamma_1} = =-2\sum_{i}\beta x_i(y_i-\beta(\gamma_0+\gamma_1 x_i))=0.
 \]
 !et
 
 
+!split
+=====   Finding the Optimal Parameters =====
+
+With these equations we can then in turn find the parameters $\beta_1$ and $\gamma_0^{1}$ and $\gamma_1^1$ as
+
+
 
 !split
 ===== Iterative Fitting, Classification, AdaBoost =====
@@ -1747,14 +1753,15 @@ The error rate of the training sample is then
 
 !bt
 \[
-\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i). 
+\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)). 
 \]
 !et
 
 The iterative procedure starts with defining a weak classifier whose
-error rate is barely better than random guessing.  Teh iterative
+error rate is barely better than random guessing.  The iterative
 procedure in boosting is to sequentially apply a the weak
-classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers $G_m(x)$.
+classification algorithm to repeatedly modified versions of the data
+producing a sequence of weak classifiers $G_m(x)$.
 
 Here we will express our  function $f(x)$ in terms of $G(x)$. That is
 !bt
@@ -1785,7 +1792,7 @@ The simplest possible cost function which leads (also simple from a computationa
 exponential cost/loss function defined as
 !bt
 \[
-C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}\exp{-(y_i(f_{m-1}(x_i)+\beta G(x_i))}.
+C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}.
 \]
 !et
 
@@ -1794,10 +1801,10 @@ This is normally done in two steps. Let us however first rewrite the cost functi
 
 !bt
 \[
-C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{-(y_i\beta G(x_i))},
+C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))},
 \]
 !et
-where we have defined $w_i^m= \exp{-(y_if_{m-1}(x_i))}$.
+where we have defined $w_i^m= \exp{(-y_if_{m-1}(x_i))}$.
 
 !split
 ===== Building up AdaBoost =====
@@ -1819,7 +1826,7 @@ We can do this by rewriting
 which can be rewritten as
 !bt
 \[
-(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{-(\beta)}\sum_{i=0}^{n-1}w_i^m=0,
+(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0,
 \]
 !et
 which leads to
@@ -1843,7 +1850,7 @@ f_m(x) = f_{m-1}(x) +\beta_m G_m(x).
 This leads to the new weights
 !bt
 \[
-w_i^{m+1} = w_i^m \exp{-(y_i\beta_m G_m(x_i))}
+w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))}
 \]
 !et
 !split
@@ -1857,10 +1864,10 @@ feature/predictor vectors
 $\bm{X}=[\bm{x}_0\bm{x}_1\dots\bm{x}_{p-1}]$. Finally, we define also a
 classifier determined by our data via a function $G(x)$. This function tells us how well we are able to classify our outputs/targets $\bm{y}$. 
 
-We can then define the misclassification error $\mathrm{err}$ as
+We have already defined the misclassification error $\mathrm{err}$ as
 !bt
 \[
-\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i),
+\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)),
 \]
 !et
 where the function $I()$ is one if we misclassify and zero if we classify correctly. 
@@ -2011,7 +2018,7 @@ plt.show()
 
 
 !split
-===== Gradient Boosting, Examples of Classification =====
+===== Gradient Boosting, Classification Example =====
 !bc pycod
 import matplotlib.pyplot as plt
 import numpy as np
@@ -2128,6 +2135,8 @@ plt.show()
 
 !split
 ===== Xgboost on the Cancer Data =====
+
+As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now. 
 !bc pycod
 
 import matplotlib.pyplot as plt