$$
-\hat{\boldsymbol{\beta}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
@@ -701,9 +701,44 @@ Inserting the above values we obtain that
$$
-\hat{\boldsymbol{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix},
+\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix},
$$
+
+
+Computing the mean squared error we obtian a value of \( 0.27 \). + +
+The code which implements this simpler case is presented after the discussion of Ridge and Lasso.
+
+
+
+
+For Ridge regression we have
+
+
+There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \).
+Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \).
+
+
+To see this, let us write the cost function for Ridge regression.
The Ridge case
+
+
+$$
+\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+$$
+
+
+Inserting the above values we obtain that
+
+
+$$
+\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{matrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
+$$
+
+
+
+Computing the mean squared error we obtian a value of \( 0.27 \). + +
+The code which implements this simpler case is presented after the discussion of Ridge and Lasso. + +
+
+
+
+For Ridge regression we have + +$$ +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +Inserting the above values we obtain that + +$$ +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{matrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +$$ + +
+There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \). +Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \). + +
+To see this, let us write the cost function for Ridge regression. +
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index 981ad013e..d117d30dd 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -112,6 +112,7 @@ div { text-align: justify; text-justify: inter-word; } ('Lasso Regression', 2, None, 'lasso-regression'), ('Yet another Example', 2, None, 'yet-another-example'), ('The OLS case', 2, None, 'the-ols-case'), + ('The Ridge case', 2, None, 'the-ridge-case'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -711,15 +712,46 @@ meaning that we have two features and two unknown parameters \( \beta_0 \) and \ For ordinary least squares (OLS) we know that the optimal solution is $$ -\hat{\boldsymbol{\beta}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ Inserting the above values we obtain that $$ -\hat{\boldsymbol{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix}, $$ +
+Computing the mean squared error we obtian a value of \( 0.27 \). + +
+The code which implements this simpler case is presented after the discussion of Ridge and Lasso. + +
+
+
+
+For Ridge regression we have + +$$ +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +Inserting the above values we obtain that + +$$ +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{matrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +$$ + +
+There is normally a constraint on the value of $\vert\vert \boldsymbol{\beta}\vert\vert_2 via the parameter \( \lambda \). +Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression with the optimal values of \( \beta \) and \( \lambda \). + +
+To see this, let us write the cost function for Ridge regression. +
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index 18bebd87b..4fbe9544c 100644 Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 771d553ac..1aa22c4c9 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -824,7 +824,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, @@ -840,7 +840,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}=\\begin{matrix}\\frac{11}{5} \\\\ 2\\end{bmatrix},\n", + "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{matrix}\\frac{11}{5} \\\\ 2\\end{bmatrix},\n", "$$" ] }, @@ -848,6 +848,50 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "Computing the mean squared error we obtian a value of $0.27$. \n", + "\n", + "The code which implements this simpler case is presented after the discussion of Ridge and Lasso.\n", + "\n", + "## The Ridge case\n", + "\n", + "For Ridge regression we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Inserting the above values we obtain that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{matrix}\\frac{11}{5+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2 via the parameter $\\lambda$.\n", + "Let us for simplicity assume now that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression with the optimal values of $\\beta$ and $\\lambda$.\n", + "\n", + "To see this, let us write the cost function for Ridge regression. \n", + "\n", + "\n", "\n", "## Linking the regression analysis with a statistical interpretation\n", "\n", diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index 0e3896a03..dd3573afa 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -454,17 +454,43 @@ For ordinary least squares (OLS) we know that the optimal solution is !bt \[ -\hat{\bm{\beta}}=\left( \bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\hat{\bm{\beta}}^{\mathrm{OLS}}=\left( \bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. \] !et Inserting the above values we obtain that !bt \[ -\hat{\bm{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix}, +\hat{\bm{\beta}}^{\mathrm{OLS}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix}, \] !et +Computing the mean squared error we obtian a value of $0.27$. + +The code which implements this simpler case is presented after the discussion of Ridge and Lasso. + +!split +===== The Ridge case ===== + +For Ridge regression we have + +!bt +\[ +\hat{\bm{\beta}}^{\mathrm{Ridge}}=\left( \bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}. +\] +!et +Inserting the above values we obtain that + +!bt +\[ +\hat{\bm{\beta}}^{\mathrm{Ridge}}=\begin{matrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +\] +!et + +There is normally a constraint on the value of $\vert\vert \bm{\beta}\vert\vert_2 via the parameter $\lambda$. +Let us for simplicity assume now that $\beta_0^2+\beta_1^2=1$ as constraint. This will allow us to find an expression with the optimal values of $\beta$ and $\lambda$. + +To see this, let us write the cost function for Ridge regression. !split