cleaning up

doc/src/week38/week38.do.txt

@@ -1355,82 +1355,7 @@ large we can experience erratic behavior.
 
 Many of these shortcomings can be alleviated by introducing
 randomness. One such method is that of Stochastic Gradient Descent
-(SGD), see below.
-
-
-
-
-
-!split
-=====  Revisiting our Logistic Regression case =====
-
-In our discussion on Logistic Regression we studied the 
-case of
-two classes, with $y_i$ either
-$0$ or $1$. Furthermore we assumed also that we have only two
-parameters $\beta$ in our fitting, that is we
-defined probabilities
-
-!bt
-\begin{align*}
-p(y_i=1|x_i,\bm{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\bm{\beta}) &= 1 - p(y_i=1|x_i,\bm{\beta}),
-\end{align*}
-!et
-where $\bm{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. 
-
-!split
-===== The equations to solve =====
-
-Our compact equations used a definition of a vector $\bm{y}$ with $n$
-elements $y_i$, an $n\times p$ matrix $\bm{X}$ which contains the
-$x_i$ values and a vector $\bm{p}$ of fitted probabilities
-$p(y_i\vert x_i,\bm{\beta})$. We rewrote in a more compact form
-the first derivative of the cost function as
-
-!bt
-\[
-\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right). 
-\]
-!et
-
-If we in addition define a diagonal matrix $\bm{W}$ with elements 
-$p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compact expression of the second derivative as 
-
-!bt
-\[
-\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} = \bm{X}^T\bm{W}\bm{X}. 
-\]
-!et
-This defines what is called  the Hessian matrix.
-
-
-
-
-
-!split
-===== Solving using Newton-Raphson's method =====
-
-If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the  matrices that define the first and second derivatives. 
-
-Our iterative scheme is then given by
-
-!bt
-\[
-\bm{\beta}^{\mathrm{new}} = \bm{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T}\right)^{-1}_{\bm{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}}\right)_{\bm{\beta}^{\mathrm{old}}},
-\]
-!et
-or in matrix form as
-
-!bt
-\[
-\bm{\beta}^{\mathrm{new}} = \bm{\beta}^{\mathrm{old}}-\left(\bm{X}^T\bm{W}\bm{X} \right)^{-1}\times \left(-\bm{X}^T(\bm{y}-\bm{p}) \right)_{\bm{\beta}^{\mathrm{old}}}.
-\]
-!et
-The right-hand side is computed with the old values of $\beta$. 
-
-If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement. 
-
+(SGD), to be discussed next week.
 
 
 !split