diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 640edc005..f00110cfe 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -1352,13 +1352,14 @@ display(XPandas) print(XPandas.std()) XPandas = (XPandas -XPandas.mean()) display(XPandas) +# This option does not include the standard deviation scaler = StandardScaler(with_std=False) scaler.fit(X) Xscaled = scaler.transform(X) display(XPandas-Xscaled)

-Small exercise: perform the standard scaling by including the standard deviation. +Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives. @@ -1556,33 +1557,36 @@ clf = skl.LinearRegression().fit(X_train_scaled, y_train) What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).

-We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by +We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by

 
$$ -\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

 
+

+The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

This means that our best model is defined as

 
$$ -\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

 

-We define now a matrix +We now define a matrix

 
$$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T. +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$

 

-It means that we can rewrite +We can rewrite

 
$$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. @@ -1591,7 +1595,7 @@ $$

The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. -We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). +We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix. @@ -1607,7 +1611,7 @@ $$

 

-The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \). +The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \). @@ -1627,7 +1631,7 @@ $$ In this case the matrix \( \boldsymbol{A} \) becomes

 
$$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$

 
@@ -1637,6 +1641,9 @@ $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$

 
+ +

+This serves also as a useful test of our codes. diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index 4ec03d491..5e25f4872 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -1450,13 +1450,14 @@ display(XPandas) print(XPandas.std()) XPandas = (XPandas -XPandas.mean()) display(XPandas) +# This option does not include the standard deviation scaler = StandardScaler(with_std=False) scaler.fit(X) Xscaled = scaler.transform(X) display(XPandas-Xscaled)

-Small exercise: perform the standard scaling by including the standard deviation. +Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.











@@ -1650,34 +1651,37 @@ clf = skl.LinearRegression().fit(X_train_scaled, y_train) What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).

-We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by +We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by $$ -\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ +

+The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

This means that our best model is defined as $$ -\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

-We define now a matrix +We now define a matrix $$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T. +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$

-It means that we can rewrite +We can rewrite $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. $$

The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. -We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). +We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.











@@ -1691,7 +1695,7 @@ $$ $$

-The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \). +The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).











@@ -1708,7 +1712,7 @@ $$

In this case the matrix \( \boldsymbol{A} \) becomes $$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$ and we have the obvious case @@ -1716,6 +1720,9 @@ $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$ +

+This serves also as a useful test of our codes. +











diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 301360694..89169d7b7 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -1455,13 +1455,14 @@ display(XPandas) print(XPandas.std()) XPandas = (XPandas -XPandas.mean()) display(XPandas) +# This option does not include the standard deviation scaler = StandardScaler(with_std=False) scaler.fit(X) Xscaled = scaler.transform(X) display(XPandas-Xscaled)

-Small exercise: perform the standard scaling by including the standard deviation. +Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.











@@ -1655,34 +1656,37 @@ clf = skl. -We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by +We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by $$ -\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ +

+The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

This means that our best model is defined as $$ -\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$

-We define now a matrix +We now define a matrix $$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T. +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$

-It means that we can rewrite +We can rewrite $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. $$

The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. -We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). +We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.











@@ -1696,7 +1700,7 @@ $$ $$

-The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \). +The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).











@@ -1713,7 +1717,7 @@ $$

In this case the matrix \( \boldsymbol{A} \) becomes $$ -\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$ and we have the obvious case @@ -1721,6 +1725,9 @@ $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$ +

+This serves also as a useful test of our codes. +











diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index c2fa6453e..bf6411588 100644 Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 292bea0b2..9542aa587 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -1663,6 +1663,7 @@ "print(XPandas.std())\n", "XPandas = (XPandas -XPandas.mean())\n", "display(XPandas)\n", + "# This option does not include the standard deviation\n", "scaler = StandardScaler(with_std=False)\n", "scaler.fit(X)\n", "Xscaled = scaler.transform(X)\n", @@ -1673,7 +1674,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Small exercise: perform the standard scaling by including the standard deviation.\n", + "Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.\n", "\n", "## Min-Max Scaling\n", "\n", @@ -1900,7 +1901,7 @@ "What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n", "\n", "\n", - "We have shown that in ordinary least squares that the optimal parameter $\\beta$ are given by" + "We have shown that in ordinary least squares the optimal parameters $\\beta$ are given by" ] }, { @@ -1908,7 +1909,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, @@ -1916,6 +1917,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n", + "\n", "This means that our best model is defined as" ] }, @@ -1924,7 +1927,7 @@ "metadata": {}, "source": [ "$$\n", - "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, @@ -1932,7 +1935,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We define now a matrix" + "We now define a matrix" ] }, { @@ -1940,7 +1943,7 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T.\n", + "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T.\n", "$$" ] }, @@ -1948,7 +1951,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "It means that we can rewrite" + "We can rewrite" ] }, { @@ -1965,7 +1968,9 @@ "metadata": {}, "source": [ "The matrix $\\boldsymbol{A}$ has the important property that $\\boldsymbol{A}^2=\\boldsymbol{A}$. This is the definition of a projection matrix.\n", - "We can then interpret that our optimal model $\\tilde{\\boldsymbol{y}}$ ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$.\n", + "We can then interpret our optimal model $\\tilde{\\boldsymbol{y}}$ as being represented by an orthogonal projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$. In our case here the matrix $\\boldsymbol{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.\n", + "\n", + "\n", "\n", "## Residual Error\n", "\n", @@ -1985,7 +1990,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal components of the space defined by the column vectors of $\\boldsymbol{X}$.\n", + "The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal component of the space defined by the column vectors of $\\boldsymbol{X}$.\n", "\n", "## Simple case\n", "\n", @@ -2013,7 +2018,7 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T)=\\boldsymbol{I},\n", + "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T)=\\boldsymbol{I},\n", "$$" ] }, @@ -2037,6 +2042,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "This serves also as a useful test of our codes. \n", + "\n", "## The singular value decomposition\n", "\n", "\n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index a163b575b..2f3fd9b54 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -999,13 +999,14 @@ print(XPandas.mean()) print(XPandas.std()) XPandas = (XPandas -XPandas.mean()) display(XPandas) +# This option does not include the standard deviation scaler = StandardScaler(with_std=False) scaler.fit(X) Xscaled = scaler.transform(X) display(XPandas-Xscaled) !ec -Small exercise: perform the standard scaling by including the standard deviation. +Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives. !split ===== Min-Max Scaling ===== @@ -1189,30 +1190,32 @@ print("R2 score for scaled data: {:.2f}".format(clf.score(X_test_scaled,y_test) What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). -We have shown that in ordinary least squares that the optimal parameter $\beta$ are given by +We have shown that in ordinary least squares the optimal parameters $\beta$ are given by !bt \[ -\hat{\bm{\beta}} = \left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T\bm{y}. +\hat{\bm{\beta}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. \] !et +The _hat_ over $\bm{\beta}$ means we have the optimal parameters after minimization of the cost function. + This means that our best model is defined as !bt \[ -\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T\bm{y}. +\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. \] !et -We define now a matrix +We now define a matrix !bt \[ -\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T. +\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T. \] !et -It means that we can rewrite +We can rewrite !bt \[ \tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{A}\bm{y}. @@ -1220,7 +1223,9 @@ It means that we can rewrite !et The matrix $\bm{A}$ has the important property that $\bm{A}^2=\bm{A}$. This is the definition of a projection matrix. -We can then interpret that our optimal model $\tilde{\bm{y}}$ ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of $\bm{y}$ onto a space defined by the column vectors of $\bm{X}$. +We can then interpret our optimal model $\tilde{\bm{y}}$ as being represented by an orthogonal projection of $\bm{y}$ onto a space defined by the column vectors of $\bm{X}$. In our case here the matrix $\bm{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix. + + !split ===== Residual Error ===== @@ -1232,7 +1237,7 @@ We have defined the residual error as \] !et -The residual errors are then the projections of $\bm{y}$ onto the orthogonal components of the space defined by the column vectors of $\bm{X}$. +The residual errors are then the projections of $\bm{y}$ onto the orthogonal component of the space defined by the column vectors of $\bm{X}$. !split ===== Simple case ===== @@ -1248,7 +1253,7 @@ If the matrix $\bm{X}$ is an orthogonal (or unitary in case of complex values) m In this case the matrix $\bm{A}$ becomes !bt \[ -\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T)=\bm{I}, +\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T)=\bm{I}, \] !et and we have the obvious case @@ -1258,7 +1263,7 @@ and we have the obvious case \] !et - +This serves also as a useful test of our codes. !split ===== The singular value decomposition =====