-
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs046.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs046.html index 7155b82a4..385a0138a 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs046.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs046.html @@ -305,7 +305,7 @@ $$
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and -\( \gamma_1 \) where determined by the Logistic Regression fitting +\( \gamma_1 \) were determined by the Logistic Regression fitting algorithm.
@@ -317,8 +317,8 @@ $$
In this case the function \( f(x) \) was replaced by the design matrix \( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \), -that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we could -simply invert a matrix and obtained the parameters \( \beta \) by +that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can +simply invert a matrix and obtain the parameters \( \beta \) by $$ \boldsymbol{\beta}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs048.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs048.html index c2a0e92e0..f8977779c 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs048.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs048.html @@ -287,10 +287,10 @@ To better understand what happens, let us develop the steps for the iterative fi For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
-This means that for every iteration, we need to optimize +This means that for every iteration \( m \), we need to optimize $$ -(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. +(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. $$
diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs049.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs049.html index 19614cf2b..f231d953e 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs049.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs049.html @@ -282,7 +282,7 @@ MathJax.Hub.Config({
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values +observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values \( \{-1,1\} \).
@@ -295,7 +295,7 @@ $$
The iterative procedure starts with defining a weak classifier whose error rate is barely better than random guessing. The iterative -procedure in boosting is to sequentially apply a the weak +procedure in boosting is to sequentially apply a weak classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers \( G_m(x) \). diff --git a/doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html b/doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html index b2a8f0543..c8fb5fedc 100644 --- a/doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html +++ b/doc/pub/DecisionTrees/html/._DecisionTrees-bs053.html @@ -285,7 +285,7 @@ With the above definitions we are now ready to set up the algorithm for AdaBoost The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
-Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here. +Using Scikit-Learn it is easy to apply the adaptive boosting algorithm, as done here.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-bs.html b/doc/pub/DecisionTrees/html/DecisionTrees-bs.html index 589bdef71..270401bc0 100644 --- a/doc/pub/DecisionTrees/html/DecisionTrees-bs.html +++ b/doc/pub/DecisionTrees/html/DecisionTrees-bs.html @@ -297,7 +297,7 @@ MathJax.Hub.Config({
-
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html b/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html index d7a0c606c..1169dea27 100644 --- a/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html +++ b/doc/pub/DecisionTrees/html/DecisionTrees-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
@@ -2072,7 +2072,7 @@ $$
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and -\( \gamma_1 \) where determined by the Logistic Regression fitting +\( \gamma_1 \) were determined by the Logistic Regression fitting algorithm.
@@ -2086,8 +2086,8 @@ $$
In this case the function \( f(x) \) was replaced by the design matrix \( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \), -that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we could -simply invert a matrix and obtained the parameters \( \beta \) by +that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can +simply invert a matrix and obtain the parameters \( \beta \) by
$$
@@ -2136,11 +2136,11 @@ To better understand what happens, let us develop the steps for the iterative fi
For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
-This means that for every iteration, we need to optimize +This means that for every iteration \( m \), we need to optimize
$$
-(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
+(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
$$
@@ -2189,7 +2189,7 @@ The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values +observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values \( \{-1,1\} \).
@@ -2204,7 +2204,7 @@ $$
The iterative procedure starts with defining a weak classifier whose error rate is barely better than random guessing. The iterative -procedure in boosting is to sequentially apply a the weak +procedure in boosting is to sequentially apply a weak classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers \( G_m(x) \). @@ -2349,7 +2349,7 @@ With the above definitions we are now ready to set up the algorithm for AdaBoost The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
@@ -2363,7 +2363,7 @@ $$
For the iterations with \( m \le 2 \) the weights are modified -individually at each steps. The obersvations which were misclassified +individually at each steps. The observations which were misclassified at iteration \( m-1 \) have a weight which is larger than those which were classified properly. As this proceeds, the observations which were difficult to classifiy correctly are given a larger influence. Each @@ -2386,7 +2386,7 @@ observations that are missed in the previous iterations.
-Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here. +Using Scikit-Learn it is easy to apply the adaptive boosting algorithm, as done here.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html index de6f5d392..627c7f756 100644 --- a/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html +++ b/doc/pub/DecisionTrees/html/DecisionTrees-solarized.html @@ -223,7 +223,7 @@ MathJax.Hub.Config({
-
@@ -2080,7 +2080,7 @@ $$
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and -\( \gamma_1 \) where determined by the Logistic Regression fitting +\( \gamma_1 \) were determined by the Logistic Regression fitting algorithm.
@@ -2092,8 +2092,8 @@ $$
In this case the function \( f(x) \) was replaced by the design matrix \( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \), -that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we could -simply invert a matrix and obtained the parameters \( \beta \) by +that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can +simply invert a matrix and obtain the parameters \( \beta \) by $$ \boldsymbol{\beta}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. @@ -2139,10 +2139,10 @@ To better understand what happens, let us develop the steps for the iterative fi For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
-This means that for every iteration, we need to optimize +This means that for every iteration \( m \), we need to optimize $$ -(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. +(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. $$
@@ -2182,7 +2182,7 @@ The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values +observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values \( \{-1,1\} \).
@@ -2195,7 +2195,7 @@ $$
The iterative procedure starts with defining a weak classifier whose error rate is barely better than random guessing. The iterative -procedure in boosting is to sequentially apply a the weak +procedure in boosting is to sequentially apply a weak classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers \( G_m(x) \). @@ -2313,7 +2313,7 @@ With the above definitions we are now ready to set up the algorithm for AdaBoost The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
-Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here. +Using Scikit-Learn it is easy to apply the adaptive boosting algorithm, as done here.
diff --git a/doc/pub/DecisionTrees/html/DecisionTrees.html b/doc/pub/DecisionTrees/html/DecisionTrees.html index 6327e297c..339c930bf 100644 --- a/doc/pub/DecisionTrees/html/DecisionTrees.html +++ b/doc/pub/DecisionTrees/html/DecisionTrees.html @@ -228,7 +228,7 @@ MathJax.Hub.Config({
-
@@ -2085,7 +2085,7 @@ $$
where \( t=\gamma_0+\gamma_1 x \) and the parameters \( \gamma_0 \) and -\( \gamma_1 \) where determined by the Logistic Regression fitting +\( \gamma_1 \) were determined by the Logistic Regression fitting algorithm.
@@ -2097,8 +2097,8 @@ $$
In this case the function \( f(x) \) was replaced by the design matrix \( \boldsymbol{X} \) and the unknown linear regression parameters \( \boldsymbol{\beta} \), -that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we could -simply invert a matrix and obtained the parameters \( \beta \) by +that is \( \boldsymbol{f}=\boldsymbol{X}\boldsymbol{\beta} \). In linear regression we can +simply invert a matrix and obtain the parameters \( \beta \) by $$ \boldsymbol{\beta}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. @@ -2144,10 +2144,10 @@ To better understand what happens, let us develop the steps for the iterative fi For simplicity we assume also that our functions \( b(x;\gamma)=1+\gamma x \).
-This means that for every iteration, we need to optimize +This means that for every iteration \( m \), we need to optimize $$ -(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. +(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. $$
@@ -2187,7 +2187,7 @@ The solution to these two equations gives us in turn \( \beta_1 \) and \( \gamma
Let us consider a binary classification problem with two outcomes \( y_i \in \{-1,1\} \) and \( i=0,1,2,\dots,n-1 \) as our set of -observations. We define a classification function \( G(x) \) which produces a prediction taking any of the two values +observations. We define a classification function \( G(x) \) which produces a prediction taking one or the other of the two values \( \{-1,1\} \).
@@ -2200,7 +2200,7 @@ $$
The iterative procedure starts with defining a weak classifier whose error rate is barely better than random guessing. The iterative -procedure in boosting is to sequentially apply a the weak +procedure in boosting is to sequentially apply a weak classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers \( G_m(x) \). @@ -2318,7 +2318,7 @@ With the above definitions we are now ready to set up the algorithm for AdaBoost The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
-Using Scikit-Learn it is easy to appply the adaptive boosting algorithm, as done here. +Using Scikit-Learn it is easy to apply the adaptive boosting algorithm, as done here.
diff --git a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb index 2015dd806..1885b3cc2 100644 --- a/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb +++ b/doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Nov 24, 2019**\n", + "Date: **Dec 5, 2019**\n", "\n", "Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -2145,7 +2145,7 @@ "metadata": {}, "source": [ "where $t=\\gamma_0+\\gamma_1 x$ and the parameters $\\gamma_0$ and\n", - "$\\gamma_1$ where determined by the Logistic Regression fitting\n", + "$\\gamma_1$ were determined by the Logistic Regression fitting\n", "algorithm.\n", "\n", "As another example, consider the cost function we defined for linear regression" @@ -2166,8 +2166,8 @@ "source": [ "In this case the function $f(x)$ was replaced by the design matrix\n", "$\\boldsymbol{X}$ and the unknown linear regression parameters $\\boldsymbol{\\beta}$,\n", - "that is $\\boldsymbol{f}=\\boldsymbol{X}\\boldsymbol{\\beta}$. In linear regression we could\n", - "simply invert a matrix and obtained the parameters $\\beta$ by" + "that is $\\boldsymbol{f}=\\boldsymbol{X}\\boldsymbol{\\beta}$. In linear regression we can \n", + "simply invert a matrix and obtain the parameters $\\beta$ by" ] }, { @@ -2214,7 +2214,7 @@ "\n", "For simplicity we assume also that our functions $b(x;\\gamma)=1+\\gamma x$. \n", "\n", - "This means that for every iteration, we need to optimize" + "This means that for every iteration $m$, we need to optimize" ] }, { @@ -2222,7 +2222,7 @@ "metadata": {}, "source": [ "$$\n", - "(\\beta_m,\\gamma_m) \\mathrm{argmin}_{\\beta,\\lambda}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2=\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(1+\\gamma x_i))^2.\n", + "(\\beta_m,\\gamma_m) = \\mathrm{argmin}_{\\beta,\\lambda}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta b(x;\\gamma))^2=\\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\\beta(1+\\gamma x_i))^2.\n", "$$" ] }, @@ -2306,7 +2306,7 @@ "## Iterative Fitting, Classification and AdaBoost\n", "\n", "Let us consider a binary classification problem with two outcomes $y_i \\in \\{-1,1\\}$ and $i=0,1,2,\\dots,n-1$ as our set of\n", - "observations. We define a classification function $G(x)$ which produces a prediction taking any of the two values \n", + "observations. We define a classification function $G(x)$ which produces a prediction taking one or the other of the two values \n", "$\\{-1,1\\}$.\n", "\n", "The error rate of the training sample is then" @@ -2327,7 +2327,7 @@ "source": [ "The iterative procedure starts with defining a weak classifier whose\n", "error rate is barely better than random guessing. The iterative\n", - "procedure in boosting is to sequentially apply a the weak\n", + "procedure in boosting is to sequentially apply a weak\n", "classification algorithm to repeatedly modified versions of the data\n", "producing a sequence of weak classifiers $G_m(x)$.\n", "\n", @@ -2565,7 +2565,7 @@ "\n", "With the above definitions we are now ready to set up the algorithm for AdaBoost.\n", "The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.\n", - "1. We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\\dots n-1$. It is to see then that $\\sum_{i=0}^{n-1}w_i = 1$.\n", + "1. We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\\dots n-1$. It is easy to see that we must have $\\sum_{i=0}^{n-1}w_i = 1$.\n", "\n", "2. We rewrite the misclassification error as" ] @@ -2585,7 +2585,7 @@ "source": [ "1. Then we start looping over all attempts at classifying, namely we start an iterative process for $m=1:M$, where $M$ is the final number of classifications. Our given classifier could for example be a plain decision tree.\n", "\n", - "a. Fit then a given classifier to the training using the weights $w_i$.\n", + "a. Fit then a given classifier to the training set using the weights $w_i$.\n", "\n", "b. Compute then $\\mathrm{err}$ and figure out which events are classified properly and which are classified wrongly.\n", "\n", @@ -2597,7 +2597,7 @@ "5. Compute the new classifier $G(x)= \\sum_{i=0}^{n-1}\\alpha_m I(y_i\\ne G(x_i)$.\n", "\n", "For the iterations with $m \\le 2$ the weights are modified\n", - "individually at each steps. The obersvations which were misclassified\n", + "individually at each steps. The observations which were misclassified\n", "at iteration $m-1$ have a weight which is larger than those which were\n", "classified properly. As this proceeds, the observations which were\n", "difficult to classifiy correctly are given a larger influence. Each\n", @@ -2608,7 +2608,7 @@ "\n", "## AdaBoost Examples\n", "\n", - "Using **Scikit-Learn** it is easy to appply the adaptive boosting algorithm, as done here." + "Using **Scikit-Learn** it is easy to apply the adaptive boosting algorithm, as done here." ] }, { diff --git a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz index b8b9833fb..a0743d665 100644 Binary files a/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz and b/doc/pub/DecisionTrees/ipynb/ipynb-DecisionTrees-src.tar.gz differ diff --git a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf index fcb8c2114..a60a2f889 100644 Binary files a/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf and b/doc/pub/DecisionTrees/pdf/DecisionTrees-minted.pdf differ diff --git a/doc/src/DecisionTrees/DecisionTrees.do.txt b/doc/src/DecisionTrees/DecisionTrees.do.txt index a6d2851e1..bf8941655 100644 --- a/doc/src/DecisionTrees/DecisionTrees.do.txt +++ b/doc/src/DecisionTrees/DecisionTrees.do.txt @@ -1709,7 +1709,7 @@ $b(x;\gamma_m)$ into the Sigmoid function !et where $t=\gamma_0+\gamma_1 x$ and the parameters $\gamma_0$ and -$\gamma_1$ where determined by the Logistic Regression fitting +$\gamma_1$ were determined by the Logistic Regression fitting algorithm. As another example, consider the cost function we defined for linear regression @@ -1721,8 +1721,8 @@ C(\bm{y},\bm{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f(x_i))^2. In this case the function $f(x)$ was replaced by the design matrix $\bm{X}$ and the unknown linear regression parameters $\bm{\beta}$, -that is $\bm{f}=\bm{X}\bm{\beta}$. In linear regression we could -simply invert a matrix and obtained the parameters $\beta$ by +that is $\bm{f}=\bm{X}\bm{\beta}$. In linear regression we can +simply invert a matrix and obtain the parameters $\beta$ by !bt \[ @@ -1757,11 +1757,11 @@ To better understand what happens, let us develop the steps for the iterative fi For simplicity we assume also that our functions $b(x;\gamma)=1+\gamma x$. -This means that for every iteration, we need to optimize +This means that for every iteration $m$, we need to optimize !bt \[ -(\beta_m,\gamma_m) \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. +(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2. \] !et @@ -1803,7 +1803,7 @@ $f_1(x) = \beta_1(1+\gamma_1x)$. Doing this $M$ times results in our final estim ===== Iterative Fitting, Classification and AdaBoost ===== Let us consider a binary classification problem with two outcomes $y_i \in \{-1,1\}$ and $i=0,1,2,\dots,n-1$ as our set of -observations. We define a classification function $G(x)$ which produces a prediction taking any of the two values +observations. We define a classification function $G(x)$ which produces a prediction taking one or the other of the two values $\{-1,1\}$. The error rate of the training sample is then @@ -1816,7 +1816,7 @@ The error rate of the training sample is then The iterative procedure starts with defining a weak classifier whose error rate is barely better than random guessing. The iterative -procedure in boosting is to sequentially apply a the weak +procedure in boosting is to sequentially apply a weak classification algorithm to repeatedly modified versions of the data producing a sequence of weak classifiers $G_m(x)$. @@ -1934,7 +1934,7 @@ where the function $I()$ is one if we misclassify and zero if we classify correc With the above definitions we are now ready to set up the algorithm for AdaBoost. The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases. -o We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\dots n-1$. It is to see then that $\sum_{i=0}^{n-1}w_i = 1$. +o We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\dots n-1$. It is easy to see that we must have $\sum_{i=0}^{n-1}w_i = 1$. o We rewrite the misclassification error as !bt \[ @@ -1942,14 +1942,14 @@ o We rewrite the misclassification error as \] !et o Then we start looping over all attempts at classifying, namely we start an iterative process for $m=1:M$, where $M$ is the final number of classifications. Our given classifier could for example be a plain decision tree. - o Fit then a given classifier to the training using the weights $w_i$. + o Fit then a given classifier to the training set using the weights $w_i$. o Compute then $\mathrm{err}$ and figure out which events are classified properly and which are classified wrongly. o Define a quantity $\alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m}$ o Set the new weights to $w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)}$. o Compute the new classifier $G(x)= \sum_{i=0}^{n-1}\alpha_m I(y_i\ne G(x_i)$. For the iterations with $m \le 2$ the weights are modified -individually at each steps. The obersvations which were misclassified +individually at each steps. The observations which were misclassified at iteration $m-1$ have a weight which is larger than those which were classified properly. As this proceeds, the observations which were difficult to classifiy correctly are given a larger influence. Each @@ -1961,7 +1961,7 @@ observations that are missed in the previous iterations. !split ===== AdaBoost Examples ===== -Using _Scikit-Learn_ it is easy to appply the adaptive boosting algorithm, as done here. +Using _Scikit-Learn_ it is easy to apply the adaptive boosting algorithm, as done here. !bc pycod from sklearn.ensemble import AdaBoostClassifier