update week35

2023-08-28 08:18:04 +02:00
parent 6622252203
commit d0201bd708
14 changed files with 1880 additions and 1448 deletions
@@ -382,7 +382,8 @@ MathJax.Hub.Config({

 <ol>
 <li> See lecture notes for week 35 at <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_self"><tt>https://compphysics.github.io/MachineLearning/doc/web/course.html</tt></a></li>
- <li> CMB sections 1.1 and 3.1</li>
+ <li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</li>
+ <li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</li>
 </ol>
 <p>
 <!-- navigation buttons at the bottom of the page -->
@@ -371,7 +371,7 @@ MathJax.Hub.Config({

 <p>Our data which we want to apply a machine learning method on, consist
 of a set of inputs \( \boldsymbol{x}^T=[x_0,x_1,x_2,\dots,x_{n-1}] \) and the
-outputs we want to model \( \boldsymbol{x}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \).
+outputs we want to model \( \boldsymbol{y}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \).
 We assume  that the output data can be represented (for a regression case) by a continuous function \( f \)
 through
 </p>
@@ -392,7 +392,7 @@ value and a variance \( \sigma^2 \).
 <p>In linear regression we approximate the unknown function with another
 continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
 some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1} \).
+\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
 </p>

 <p>Last week we introduced the so-called design matrix in order to define
@@ -383,8 +383,8 @@ $$

 <p>where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
 till now we have treated \( y_i \) as the exact value. Normally, the
-response (dependent or outcome) variable \( y_i \) the outcome of a
-numerical experiment or another type of experiment and is thus only an
+response (dependent or outcome) variable \( y_i \) is the outcome of a
+numerical experiment or another type of experiment and could thus be treated itself as an
 approximation to the true value. It is then always accompanied by an
 error estimate, often limited to a statistical error estimate given by
 the standard deviation discussed earlier. In the discussion here we
@@ -407,9 +407,9 @@ $$
 \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, 
 $$

-<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \))</p>
+<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
+\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
 $$


@@ -373,7 +373,7 @@ MathJax.Hub.Config({
 <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
 <p>We can rewrite, see the derivations below, </p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
+\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
 $$

 <p>as</p>
@@ -383,7 +383,7 @@ $$
 We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
 </p>

-<p>The mean squared error is scalar and if we use the results from the last example, we define a new vector</p>
+<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
 $$
 \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
 $$
@@ -403,7 +403,7 @@ $$
 \frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
 $$

-<p>where we ued the results from example two. Inserting the last expression we obtain</p>
+<p>where we used the result from example two above. Inserting the last expression we obtain</p>
 $$
 \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
 $$
@@ -389,9 +389,9 @@ $$
 <p>For ordinary least squares, it is inversely proportional (derivation
 next week) with the variance of the optimal parameters
 \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
-(beside \( 1/n \)) equal to the covariance matrix. It plays also a very
+(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
 important role in optmization algorithms and Principal Component
-Analysis as a way to reduce the dimensionality of a machine learning
+Analysis as a way to reduce the dimensionality of a machine learning/data analysis
 problem.
 </p>

@@ -384,7 +384,7 @@ target/output variables when all other predictors are set to zero.
 Thus, if we cannot assume that the expected outputs/targets are zero
 when all predictors are zero (the columns in the design matrix), it
 may be a bad idea to implement a model which penalizes the intercept.
-Furthermore, in for example Ridge and Lasso regression, the default solutions
+Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions
 from the library <b>Scikit-Learn</b> (when not shrinking \( \beta_0 \)) for the unknown parameters
 \( \boldsymbol{\beta} \), are derived&#160;under the assumption that both \( \boldsymbol{y} \) and
 \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values.
@@ -486,12 +486,12 @@ when all our features are zero and our function crosses the \( y \)-axis (for a
 </p>

 <p>Printing the MSE, we see first that both methods give the same MSE, as
-they should.  However, when we move to for example Ridge regression,
+they should.  However, when we move to for example Ridge regression (discussed next week),
 the way we treat the intercept may give a larger or smaller MSE,
 meaning that the MSE can be penalized by the value of the
 intercept. Not including the intercept in the fit, means that the
 regularization term does not include \( \beta_0 \). For different values
-of \( \lambda \), this may lead to differeing MSE values. 
+of \( \lambda \), this may lead to differing MSE values. 
 </p>

 <p>To remind the reader, the regularization term, with the intercept in Ridge regression is given by</p>
@@ -509,7 +509,14 @@ $$
 \lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
 $$

-<p>It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.</p>
+<p>It means that, when scaling the design matrix and the outputs/targets,
+by subtracting the mean values, we have an optimization problem which
+is not penalized by the intercept. The MSE value can then be smaller
+since it focuses only on the remaining quantities. If we however bring
+back the intercept, we will get an MSE which then contains the
+intercept. This becomes more important when we discuss Ridge and Lasso
+regression next week.
+</p>

 <p>
 <!-- navigation buttons at the bottom of the page -->
@@ -211,7 +211,8 @@ MathJax.Hub.Config({

 <ol>
 <p><li> See lecture notes for week 35 at <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/web/course.html</tt></a></li>
- <p><li> CMB sections 1.1 and 3.1</li>
+ <p><li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</li>
+ <p><li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</li>
 </ol>
 </section>

@@ -243,7 +244,7 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a

 <p>Our data which we want to apply a machine learning method on, consist
 of a set of inputs \( \boldsymbol{x}^T=[x_0,x_1,x_2,\dots,x_{n-1}] \) and the
-outputs we want to model \( \boldsymbol{x}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \).
+outputs we want to model \( \boldsymbol{y}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \).
 We assume  that the output data can be represented (for a regression case) by a continuous function \( f \)
 through
 </p>
@@ -268,7 +269,7 @@ value and a variance \( \sigma^2 \).
 <p>In linear regression we approximate the unknown function with another
 continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
 some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1} \).
+\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
 </p>

 <p>Last week we introduced the so-called design matrix in order to define
@@ -342,8 +343,8 @@ $$

 <p>where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
 till now we have treated \( y_i \) as the exact value. Normally, the
-response (dependent or outcome) variable \( y_i \) the outcome of a
-numerical experiment or another type of experiment and is thus only an
+response (dependent or outcome) variable \( y_i \) is the outcome of a
+numerical experiment or another type of experiment and could thus be treated itself as an
 approximation to the true value. It is then always accompanied by an
 error estimate, often limited to a statistical error estimate given by
 the standard deviation discussed earlier. In the discussion here we
@@ -372,10 +373,10 @@ $$
 $$
 <p>&nbsp;<br>

-<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \))</p>
+<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
 <p>&nbsp;<br>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
+\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
 $$
 <p>&nbsp;<br>
 </section>
@@ -388,7 +389,7 @@ $$
 <p>We can rewrite, see the derivations below, </p>
 <p>&nbsp;<br>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
+\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
 $$
 <p>&nbsp;<br>

@@ -665,7 +666,7 @@ $$
 We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
 </p>

-<p>The mean squared error is scalar and if we use the results from the last example, we define a new vector</p>
+<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
 <p>&nbsp;<br>
 $$
 \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
@@ -693,7 +694,7 @@ $$
 $$
 <p>&nbsp;<br>

-<p>where we ued the results from example two. Inserting the last expression we obtain</p>
+<p>where we used the result from example two above. Inserting the last expression we obtain</p>
 <p>&nbsp;<br>
 $$
 \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
@@ -758,9 +759,9 @@ $$
 <p>For ordinary least squares, it is inversely proportional (derivation
 next week) with the variance of the optimal parameters
 \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
-(beside \( 1/n \)) equal to the covariance matrix. It plays also a very
+(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
 important role in optmization algorithms and Principal Component
-Analysis as a way to reduce the dimensionality of a machine learning
+Analysis as a way to reduce the dimensionality of a machine learning/data analysis
 problem.
 </p>

@@ -1537,7 +1538,7 @@ target/output variables when all other predictors are set to zero.
 Thus, if we cannot assume that the expected outputs/targets are zero
 when all predictors are zero (the columns in the design matrix), it
 may be a bad idea to implement a model which penalizes the intercept.
-Furthermore, in for example Ridge and Lasso regression, the default solutions
+Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions
 from the library <b>Scikit-Learn</b> (when not shrinking \( \beta_0 \)) for the unknown parameters
 \( \boldsymbol{\beta} \), are derived&#160;under the assumption that both \( \boldsymbol{y} \) and
 \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values.
@@ -1865,12 +1866,12 @@ when all our features are zero and our function crosses the \( y \)-axis (for a
 </p>

 <p>Printing the MSE, we see first that both methods give the same MSE, as
-they should.  However, when we move to for example Ridge regression,
+they should.  However, when we move to for example Ridge regression (discussed next week),
 the way we treat the intercept may give a larger or smaller MSE,
 meaning that the MSE can be penalized by the value of the
 intercept. Not including the intercept in the fit, means that the
 regularization term does not include \( \beta_0 \). For different values
-of \( \lambda \), this may lead to differeing MSE values. 
+of \( \lambda \), this may lead to differing MSE values. 
 </p>

 <p>To remind the reader, the regularization term, with the intercept in Ridge regression is given by</p>
@@ -1894,7 +1895,14 @@ $$
 $$
 <p>&nbsp;<br>

-<p>It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.</p>
+<p>It means that, when scaling the design matrix and the outputs/targets,
+by subtracting the mean values, we have an optimization problem which
+is not penalized by the intercept. The MSE value can then be smaller
+since it focuses only on the remaining quantities. If we however bring
+back the intercept, we will get an MSE which then contains the
+intercept. This becomes more important when we discuss Ridge and Lasso
+regression next week.
+</p>
 </section>

 <section>
@@ -332,7 +332,8 @@ MathJax.Hub.Config({

 <ol>
 <li> See lecture notes for week 35 at <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/web/course.html</tt></a></li>
- <li> CMB sections 1.1 and 3.1</li>
+ <li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</li>
+ <li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</li>
 </ol>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
 <h2 id="why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week </h2>
@@ -360,7 +361,7 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a

 <p>Our data which we want to apply a machine learning method on, consist
 of a set of inputs \( \boldsymbol{x}^T=[x_0,x_1,x_2,\dots,x_{n-1}] \) and the
-outputs we want to model \( \boldsymbol{x}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \).
+outputs we want to model \( \boldsymbol{y}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \).
 We assume  that the output data can be represented (for a regression case) by a continuous function \( f \)
 through
 </p>
@@ -381,7 +382,7 @@ value and a variance \( \sigma^2 \).
 <p>In linear regression we approximate the unknown function with another
 continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
 some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1} \).
+\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
 </p>

 <p>Last week we introduced the so-called design matrix in order to define
@@ -441,8 +442,8 @@ $$

 <p>where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
 till now we have treated \( y_i \) as the exact value. Normally, the
-response (dependent or outcome) variable \( y_i \) the outcome of a
-numerical experiment or another type of experiment and is thus only an
+response (dependent or outcome) variable \( y_i \) is the outcome of a
+numerical experiment or another type of experiment and could thus be treated itself as an
 approximation to the true value. It is then always accompanied by an
 error estimate, often limited to a statistical error estimate given by
 the standard deviation discussed earlier. In the discussion here we
@@ -465,9 +466,9 @@ $$
 \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, 
 $$

-<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \))</p>
+<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
+\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
 $$


@@ -478,7 +479,7 @@ $$
 <p>
 <p>We can rewrite, see the derivations below, </p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
+\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
 $$

 <p>as</p>
@@ -707,7 +708,7 @@ $$
 We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
 </p>

-<p>The mean squared error is scalar and if we use the results from the last example, we define a new vector</p>
+<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
 $$
 \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
 $$
@@ -727,7 +728,7 @@ $$
 \frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
 $$

-<p>where we ued the results from example two. Inserting the last expression we obtain</p>
+<p>where we used the result from example two above. Inserting the last expression we obtain</p>
 $$
 \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
 $$
@@ -778,9 +779,9 @@ $$
 <p>For ordinary least squares, it is inversely proportional (derivation
 next week) with the variance of the optimal parameters
 \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
-(beside \( 1/n \)) equal to the covariance matrix. It plays also a very
+(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
 important role in optmization algorithms and Principal Component
-Analysis as a way to reduce the dimensionality of a machine learning
+Analysis as a way to reduce the dimensionality of a machine learning/data analysis
 problem.
 </p>

@@ -1538,7 +1539,7 @@ target/output variables when all other predictors are set to zero.
 Thus, if we cannot assume that the expected outputs/targets are zero
 when all predictors are zero (the columns in the design matrix), it
 may be a bad idea to implement a model which penalizes the intercept.
-Furthermore, in for example Ridge and Lasso regression, the default solutions
+Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions
 from the library <b>Scikit-Learn</b> (when not shrinking \( \beta_0 \)) for the unknown parameters
 \( \boldsymbol{\beta} \), are derived&#160;under the assumption that both \( \boldsymbol{y} \) and
 \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values.
@@ -1837,12 +1838,12 @@ when all our features are zero and our function crosses the \( y \)-axis (for a
 </p>

 <p>Printing the MSE, we see first that both methods give the same MSE, as
-they should.  However, when we move to for example Ridge regression,
+they should.  However, when we move to for example Ridge regression (discussed next week),
 the way we treat the intercept may give a larger or smaller MSE,
 meaning that the MSE can be penalized by the value of the
 intercept. Not including the intercept in the fit, means that the
 regularization term does not include \( \beta_0 \). For different values
-of \( \lambda \), this may lead to differeing MSE values. 
+of \( \lambda \), this may lead to differing MSE values. 
 </p>

 <p>To remind the reader, the regularization term, with the intercept in Ridge regression is given by</p>
@@ -1860,7 +1861,14 @@ $$
 \lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
 $$

-<p>It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.</p>
+<p>It means that, when scaling the design matrix and the outputs/targets,
+by subtracting the mean values, we have an optimization problem which
+is not penalized by the intercept. The MSE value can then be smaller
+since it focuses only on the remaining quantities. If we however bring
+back the intercept, we will get an MSE which then contains the
+intercept. This becomes more important when we discuss Ridge and Lasso
+regression next week.
+</p>

 <!-- !split  -->
 <h2 id="the-boston-housing-data-example">The Boston housing data example </h2>
@@ -409,7 +409,8 @@ MathJax.Hub.Config({

 <ol>
 <li> See lecture notes for week 35 at <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/web/course.html</tt></a></li>
- <li> CMB sections 1.1 and 3.1</li>
+ <li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</li>
+ <li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</li>
 </ol>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
 <h2 id="why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week </h2>
@@ -437,7 +438,7 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a

 <p>Our data which we want to apply a machine learning method on, consist
 of a set of inputs \( \boldsymbol{x}^T=[x_0,x_1,x_2,\dots,x_{n-1}] \) and the
-outputs we want to model \( \boldsymbol{x}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \).
+outputs we want to model \( \boldsymbol{y}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \).
 We assume  that the output data can be represented (for a regression case) by a continuous function \( f \)
 through
 </p>
@@ -458,7 +459,7 @@ value and a variance \( \sigma^2 \).
 <p>In linear regression we approximate the unknown function with another
 continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
 some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1} \).
+\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
 </p>

 <p>Last week we introduced the so-called design matrix in order to define
@@ -518,8 +519,8 @@ $$

 <p>where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
 till now we have treated \( y_i \) as the exact value. Normally, the
-response (dependent or outcome) variable \( y_i \) the outcome of a
-numerical experiment or another type of experiment and is thus only an
+response (dependent or outcome) variable \( y_i \) is the outcome of a
+numerical experiment or another type of experiment and could thus be treated itself as an
 approximation to the true value. It is then always accompanied by an
 error estimate, often limited to a statistical error estimate given by
 the standard deviation discussed earlier. In the discussion here we
@@ -542,9 +543,9 @@ $$
 \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, 
 $$

-<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \))</p>
+<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
+\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
 $$


@@ -555,7 +556,7 @@ $$
 <p>
 <p>We can rewrite, see the derivations below, </p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
+\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
 $$

 <p>as</p>
@@ -784,7 +785,7 @@ $$
 We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
 </p>

-<p>The mean squared error is scalar and if we use the results from the last example, we define a new vector</p>
+<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
 $$
 \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
 $$
@@ -804,7 +805,7 @@ $$
 \frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
 $$

-<p>where we ued the results from example two. Inserting the last expression we obtain</p>
+<p>where we used the result from example two above. Inserting the last expression we obtain</p>
 $$
 \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
 $$
@@ -855,9 +856,9 @@ $$
 <p>For ordinary least squares, it is inversely proportional (derivation
 next week) with the variance of the optimal parameters
 \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
-(beside \( 1/n \)) equal to the covariance matrix. It plays also a very
+(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
 important role in optmization algorithms and Principal Component
-Analysis as a way to reduce the dimensionality of a machine learning
+Analysis as a way to reduce the dimensionality of a machine learning/data analysis
 problem.
 </p>

@@ -1615,7 +1616,7 @@ target/output variables when all other predictors are set to zero.
 Thus, if we cannot assume that the expected outputs/targets are zero
 when all predictors are zero (the columns in the design matrix), it
 may be a bad idea to implement a model which penalizes the intercept.
-Furthermore, in for example Ridge and Lasso regression, the default solutions
+Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions
 from the library <b>Scikit-Learn</b> (when not shrinking \( \beta_0 \)) for the unknown parameters
 \( \boldsymbol{\beta} \), are derived&#160;under the assumption that both \( \boldsymbol{y} \) and
 \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values.
@@ -1914,12 +1915,12 @@ when all our features are zero and our function crosses the \( y \)-axis (for a
 </p>

 <p>Printing the MSE, we see first that both methods give the same MSE, as
-they should.  However, when we move to for example Ridge regression,
+they should.  However, when we move to for example Ridge regression (discussed next week),
 the way we treat the intercept may give a larger or smaller MSE,
 meaning that the MSE can be penalized by the value of the
 intercept. Not including the intercept in the fit, means that the
 regularization term does not include \( \beta_0 \). For different values
-of \( \lambda \), this may lead to differeing MSE values. 
+of \( \lambda \), this may lead to differing MSE values. 
 </p>

 <p>To remind the reader, the regularization term, with the intercept in Ridge regression is given by</p>
@@ -1937,7 +1938,14 @@ $$
 \lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
 $$

-<p>It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.</p>
+<p>It means that, when scaling the design matrix and the outputs/targets,
+by subtracting the mean values, we have an optimization problem which
+is not penalized by the intercept. The MSE value can then be smaller
+since it focuses only on the remaining quantities. If we however bring
+back the intercept, we will get an MSE which then contains the
+intercept. This becomes more important when we discuss Ridge and Lasso
+regression next week.
+</p>

 <!-- !split  -->
 <h2 id="the-boston-housing-data-example">The Boston housing data example </h2>
@@ -19,7 +19,9 @@ o Thursday: Ridge and Lasso regression and Singular Value Decomposition
 === Reading recommendations: ===

  o See lecture notes for week 35 at URL:"https://compphysics.github.io/MachineLearning/doc/web/course.html"
-  o CMB sections 1.1 and 3.1
+  o Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics
+  o Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4
+  



@@ -1051,7 +1053,7 @@ target/output variables when all other predictors are set to zero.
 Thus, if we cannot assume that the expected outputs/targets are zero
 when all predictors are zero (the columns in the design matrix), it
 may be a bad idea to implement a model which penalizes the intercept.
-Furthermore, in for example Ridge and Lasso regression, the default solutions
+Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions
 from the library _Scikit-Learn_ (when not shrinking $\beta_0$) for the unknown parameters
 $\bm{\beta}$, are derived under the assumption that both $\bm{y}$ and
 $\bm{X}$ are zero centered, that is we subtract the mean values.
@@ -1327,12 +1329,12 @@ The intercept is the value of our output/target variable
 when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). 

 Printing the MSE, we see first that both methods give the same MSE, as
-they should.  However, when we move to for example Ridge regression,
+they should.  However, when we move to for example Ridge regression (discussed next week),
 the way we treat the intercept may give a larger or smaller MSE,
 meaning that the MSE can be penalized by the value of the
 intercept. Not including the intercept in the fit, means that the
 regularization term does not include $\beta_0$. For different values
-of $\lambda$, this may lead to differeing MSE values. 
+of $\lambda$, this may lead to differing MSE values. 

 To remind the reader, the regularization term, with the intercept in Ridge regression is given by
 !bt
@@ -1354,7 +1356,13 @@ For Lasso regression we have
 \]
 !et

-It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.
+It means that, when scaling the design matrix and the outputs/targets,
+by subtracting the mean values, we have an optimization problem which
+is not penalized by the intercept. The MSE value can then be smaller
+since it focuses only on the remaining quantities. If we however bring
+back the intercept, we will get an MSE which then contains the
+intercept. This becomes more important when we discuss Ridge and Lasso
+regression next week.