From b605ae446106c3f1a03cc70a2758fd2e06d16dc6 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Thu, 2 Sep 2021 16:24:20 +0200 Subject: [PATCH] added more material to week 35 --- doc/pub/week35/html/week35-reveal.html | 27 ++++++++++++------- doc/pub/week35/html/week35-solarized.html | 27 ++++++++++++------- doc/pub/week35/html/week35.html | 27 ++++++++++++------- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 191 -> 192 bytes doc/pub/week35/ipynb/week35.ipynb | 27 ++++++++++++------- doc/src/week35/week35.do.txt | 27 +++++++++++-------- 6 files changed, 84 insertions(+), 51 deletions(-) diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 640edc005..f00110cfe 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -1352,13 +1352,14 @@ display(XPandas) print(XPandas.std()) XPandas = (XPandas -XPandas.mean()) display(XPandas) +# This option does not include the standard deviation scaler = StandardScaler(with_std=False) scaler.fit(X) Xscaled = scaler.transform(X) display(XPandas-Xscaled)

-Small exercise: perform the standard scaling by including the standard deviation. +Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives. @@ -1556,33 +1557,36 @@ clf = skl.LinearRegression().fit(X_train_scaled, y_train) What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).

-We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by +We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by

 
$$ -\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

 
+

+The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

This means that our best model is defined as

 
$$ -\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

 

-We define now a matrix +We now define a matrix

 
$$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T. +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$

 

-It means that we can rewrite +We can rewrite

 
$$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. @@ -1591,7 +1595,7 @@ $$

The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. -We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). +We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix. @@ -1607,7 +1611,7 @@ $$

 

-The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \). +The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \). @@ -1627,7 +1631,7 @@ $$ In this case the matrix \( \boldsymbol{A} \) becomes

 
$$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$

 
@@ -1637,6 +1641,9 @@ $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$

 
+ +

+This serves also as a useful test of our codes. diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index 4ec03d491..5e25f4872 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -1450,13 +1450,14 @@ display(XPandas) print(XPandas.std()) XPandas = (XPandas -XPandas.mean()) display(XPandas) +# This option does not include the standard deviation scaler = StandardScaler(with_std=False) scaler.fit(X) Xscaled = scaler.transform(X) display(XPandas-Xscaled)

-Small exercise: perform the standard scaling by including the standard deviation. +Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.











@@ -1650,34 +1651,37 @@ clf = skl.LinearRegression().fit(X_train_scaled, y_train) What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).

-We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by +We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by $$ -\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ +

+The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

This means that our best model is defined as $$ -\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

-We define now a matrix +We now define a matrix $$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T. +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$

-It means that we can rewrite +We can rewrite $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. $$

The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. -We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). +We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.











@@ -1691,7 +1695,7 @@ $$ $$

-The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \). +The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).











@@ -1708,7 +1712,7 @@ $$

In this case the matrix \( \boldsymbol{A} \) becomes $$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$ and we have the obvious case @@ -1716,6 +1720,9 @@ $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$ +

+This serves also as a useful test of our codes. +











diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 301360694..89169d7b7 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -1455,13 +1455,14 @@ display(XPandas) print(XPandas.std()) XPandas = (XPandas -XPandas.mean()) display(XPandas) +# This option does not include the standard deviation scaler = StandardScaler(with_std=False) scaler.fit(X) Xscaled = scaler.transform(X) display(XPandas-Xscaled)

-Small exercise: perform the standard scaling by including the standard deviation. +Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.











@@ -1655,34 +1656,37 @@ clf = skl. -We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by +We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by $$ -\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ +

+The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

This means that our best model is defined as $$ -\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

-We define now a matrix +We now define a matrix $$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T. +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$

-It means that we can rewrite +We can rewrite $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. $$

The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. -We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). +We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.











@@ -1696,7 +1700,7 @@ $$ $$

-The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \). +The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).











@@ -1713,7 +1717,7 @@ $$

In this case the matrix \( \boldsymbol{A} \) becomes $$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$ and we have the obvious case @@ -1721,6 +1725,9 @@ $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$ +

+This serves also as a useful test of our codes. +











diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index c2fa6453e2efd39210c444ec67147a3e0fd2c454..bf641158850926ae81d01e2a7ec60c29dde42e27 100644 GIT binary patch literal 192 zcmV;x06+g9iwFRS-Y{VR1MSaC3WG2Z24L5oVoo558eJFa!bPE!ULfLFQ)^M literal 191 zcmV;w06_mAiwFRr*)U-M1MSaC3c@fD2H>uHia9|^+N7=pyKo_hc!88oZLCddlA^u6 zeSoeMH${Ye4E%%)!^|ODZ+2Pa?k-piAtYrC#$2U*N)pfS38fq;qeF3)G75wyJZ1rq ztP^HfrK!%SEUI^XLt9yXm@}UO&-@d|N?O?Ndsk@%N;}N8uHlAQCp?mEuW~3f t+VKkvUOQ+IYaS2