update week 35

doc/src/week35/week35.do.txt

@@ -503,6 +503,33 @@ matrices as upper case boldfaced letters.
 \]
 !et
 
+!split
+===== Meet the Hessian Matrix =====
+
+A very important matrix we will meet again and again in Machine Learning is the Hessian.
+It is given by the second derivative of the cost function with respect to the parameter $\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
+
+!bt
+\[
+\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{\partial}{\partial \bm{\beta}^T}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.  
+\]
+!et
+
+
+The Hessian matrix plays an important role and is defined here as
+
+!bt
+\[
+\bm{H}=\bm{X}^T\bm{X}.
+\]
+!et
+
+
+For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
+$\hat{\bm{\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
+
+_Linear algebra question:_  Can we use the Hessian matrix to say something about properties of the cost function (our optmization  problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). 
+
 !split
 ===== Interpretations and optimizing our parameters  =====
 !bblock
@@ -1483,6 +1510,22 @@ inversion algorithm for matrix inversion with $\bm{X}^T\bm{X}$ results
 in the program terminating due to a singular matrix.
 
 
+!split
+=====  Note about SVD Calculations =====
+
+The $U$, $S$, and $V$ matrices returned from the _svd()_ function
+cannot be multiplied directly.
+
+As you can see from the code, the $S$
+vector must be converted into a diagonal matrix. This may cause a 
+as
+the size of the matrices do not fit the rules of matrix
+multiplication, where the number of columns in a matrix must match the
+number of rows in the subsequent matrix.
+
+If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
+
+
 
 !split
 ===== Mathematical Properties =====