update on week 35 lectures

doc/src/week35/week35.do.txt

@@ -512,12 +512,15 @@ matrices as upper case boldfaced letters.
 !split
 ===== Meet the Hessian Matrix =====
 
-A very important matrix we will meet again and again in Machine Learning is the Hessian.
-It is given by the second derivative of the cost function with respect to the parameter $\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
+A very important matrix we will meet again and again in Machine
+Learning is the Hessian.  It is given by the second derivative of the
+cost function with respect to the parameter $\beta$. Using the above
+expression for derivatives of vectors and matrices, we find that the
+second derivative of the cost function is,
 
 !bt
 \[
-\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{\partial}{\partial \bm{\beta}^T}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.  
+\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{\partial}{\partial \bm{\beta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.  
 \]
 !et
 
@@ -531,8 +534,13 @@ The Hessian matrix plays an important role and is defined here as
 !et
 
 
-For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
-$\hat{\bm{\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
+For ordinary least squares, it is inversely proportional (derivation
+next week) with the variance of the optimal parameters
+$\hat{\bm{\beta}}$. Furthermore, we will see later this week that is
+(beside $1/n$) equal to the covariance matrix. It plays also a very
+important role in optmization algorithms and Principal Component
+Analysis as a way to reduce the dimensionality of a machine learning
+problem.
 
 _Linear algebra question:_  Can we use the Hessian matrix to say something about properties of the cost function (our optmization  problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). 
 
@@ -1575,11 +1583,11 @@ Our starting point is our design matrix $\bm{X}$ of dimension $n\times p$
 \[
 \bm{X}=
 \begin{bmatrix}
-x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
-x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
-x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
-\dots& \dots &\dots& \dots & \dots &\dots\\
-x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
+x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
+x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
+x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
+\dots& \dots &\dots & \dots & \dots &\dots\\
+x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
 \end{bmatrix}
 \]
 !et
@@ -1646,7 +1654,7 @@ contains only the singular values.   Note also (and we will use this below) that
 
 !bt
 \[
-\bm{\Sigma}^T\bm{\sigma}=
+\bm{\Sigma}^T\bm{\Sigma}=
 \begin{bmatrix}
 4& 0 \\
 0 & 1 \\
@@ -1656,7 +1664,7 @@ contains only the singular values.   Note also (and we will use this below) that
 which is a $2\times 2 $ matrix while
 !bt
 \[
-\bm{\Sigma}\bm{\sigma}^T=
+\bm{\Sigma}\bm{\Sigma}^T=
 \begin{bmatrix}
 4& 0 & 0\\
 0 & 1 & 0\\
@@ -1730,7 +1738,7 @@ that belong to $i>p-1$, give all zeros when we perform the multiplications. This
 It means that the ordinary least square model (with the optimal parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\bm{y}$ by the vectors of the matrix $\bm{U}$.
 
 !split
-===== Further properties (important gems for our analysis) =====
+===== Further properties (important for our analyses later) =====
 
 Let us study again $\bm{X}^T\bm{X}$ in terms of our SVD,
 !bt
@@ -1774,6 +1782,46 @@ with eigenvalues given by the singular values squared, that is
 \]
 !et
 
+_Important note_: we have defined our design matrix $\bm{X}$ to be an
+$n\times p$ matrix. In most supervised learning cases we have that $n
+\ge p$, and quite often we have $n >> p$. The number of columns will
+always refer to the number of features in our data set, while the
+number of rows represents the number of data inputs. Note that in
+other texts you may find the opposite notation. This has consequences
+for the definition of for example the covariance matrix.
+
+
+!split
+===== Meet the Covariance Matrix =====
+
+
+Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
+
+We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have
+
+!bt
+\[
+\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}.  
+\]
+!et
+
+
+The Hessian matrix plays an important role and is defined here as
+
+!bt
+\[
+\bm{H}=\bm{X}^T\bm{X}.
+\]
+!et
+
+
+The Hessian matrix for ordinary least squares is also proportional to
+the covariance matrix. This means also that we can use the SVD to find
+the eigenvalues of the covariance matrix and the Hessian matrix in
+terms of the singular values.
+
+
+