update week 35

doc/src/week35/week35.do.txt

@@ -1784,11 +1784,13 @@ with eigenvalues given by the singular values squared, that is
 
 _Important note_: we have defined our design matrix $\bm{X}$ to be an
 $n\times p$ matrix. In most supervised learning cases we have that $n
-\ge p$, and quite often we have $n >> p$. The number of columns will
+\ge p$, and quite often we have $n >> p$. For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix $\bm{X}^T\bm{X}$ which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations).
+
+In our lectures, the number of columns will
 always refer to the number of features in our data set, while the
 number of rows represents the number of data inputs. Note that in
 other texts you may find the opposite notation. This has consequences
-for the definition of for example the covariance matrix.
+for the definition of for example the covariance matrix and its relation to the SVD.
 
 
 !split
@@ -1797,16 +1799,18 @@ for the definition of for example the covariance matrix.
 
 Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
 
-We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have
+We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary
+least squares is proportional to the second derivative of the cost
+function, that is we have
 
 !bt
 \[
 \frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}.  
 \]
 !et
+This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
 
-
-The Hessian matrix plays an important role and is defined here as
+The Hessian matrix plays an important role and is defined in this course as
 
 !bt
 \[
@@ -1818,7 +1822,7 @@ The Hessian matrix plays an important role and is defined here as
 The Hessian matrix for ordinary least squares is also proportional to
 the covariance matrix. This means also that we can use the SVD to find
 the eigenvalues of the covariance matrix and the Hessian matrix in
-terms of the singular values.
+terms of the singular values.   Let us develop these arguments, as they will play an important role in our machine learning studies.
 
 
 !split
@@ -1857,6 +1861,17 @@ we can rewrite the covariance matrix as
 \]
 !et
 
+_Note:_ we have used $1/n$ in the above definitions of the *sample* variance and covariance. We assume then that we can calculate the exact mean value. 
+What you will find in essentially all statistics texts are equations
+with a factor $1/(n-1)$. This is called "Bessel's correction":"https://mathworld.wolfram.com/BesselsCorrection.html". This
+method corrects the bias in the estimation of the population variance
+and covariance. It also partially corrects the bias in the estimation
+of the population standard deviation. If you use a library like
+_Scikit-Learn_ or _nunmpy's_ function calculate the covariance, this
+quantity will be computed with a factor $1/(n-1)$.
+
+
+
 The covariance takes values between zero and infinity and may thus
 lead to problems with loss of numerical precision for particularly
 large values. It is common to scale the covariance matrix by