+Let us assume we have a data set with outputs/targets given by the vector + +
+$$
+\boldsymbol{y}=\begin{matrix}4 \\ 2 \\3\end{bmatrix},
+$$
+
+
+and our inputs as a \( 3\times 2 \) design matrix
+
+$$
+\boldsymbol{X}=\begin{matrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
+$$
+
+
+meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.
+
+For ordinary least squares (OLS) we know that the optimal solution is + +
+$$
+\hat{\boldsymbol{\beta}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+$$
+
+
+Inserting the above values we obtain that
+
+
+$$
+\hat{\boldsymbol{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix},
+$$
+
+
+
+
+
+Let us assume we have a data set with outputs/targets given by the vector + +$$ +\boldsymbol{y}=\begin{matrix}4 \\ 2 \\3\end{bmatrix}, +$$ + +and our inputs as a \( 3\times 2 \) design matrix +$$ +\boldsymbol{X}=\begin{matrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix}, +$$ + +meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression. + +
+
+
+
+For ordinary least squares (OLS) we know that the optimal solution is + +$$ +\hat{\boldsymbol{\beta}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +Inserting the above values we obtain that + +$$ +\hat{\boldsymbol{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix}, +$$ +
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index 6c4b38b04..981ad013e 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -110,6 +110,8 @@ div { text-align: justify; text-justify: inter-word; } 'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'), ('Ridge Regression', 2, None, 'ridge-regression'), ('Lasso Regression', 2, None, 'lasso-regression'), + ('Yet another Example', 2, None, 'yet-another-example'), + ('The OLS case', 2, None, 'the-ols-case'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -681,6 +683,43 @@ Plotting these results (figure to come) shows clearly that Lasso regression supp We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \). +
+
+
+
+Let us assume we have a data set with outputs/targets given by the vector + +$$ +\boldsymbol{y}=\begin{matrix}4 \\ 2 \\3\end{bmatrix}, +$$ + +and our inputs as a \( 3\times 2 \) design matrix +$$ +\boldsymbol{X}=\begin{matrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix}, +$$ + +meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression. + +
+
+
+
+For ordinary least squares (OLS) we know that the optimal solution is + +$$ +\hat{\boldsymbol{\beta}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +Inserting the above values we obtain that + +$$ +\hat{\boldsymbol{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix}, +$$ +
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index 0fb55f793..18bebd87b 100644 Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 1b762c8ae..771d553ac 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -778,7 +778,76 @@ "We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\\beta$.\n", "\n", "\n", + "## Yet another Example\n", "\n", + "Let us assume we have a data set with outputs/targets given by the vector" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{y}=\\begin{matrix}4 \\\\ 2 \\\\3\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and our inputs as a $3\\times 2$ design matrix" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}=\\begin{matrix}2 & 0\\\\ 0 & 1 \\\\ 1 & 0\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n", + "\n", + "## The OLS case\n", + "\n", + "For ordinary least squares (OLS) we know that the optimal solution is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\beta}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Inserting the above values we obtain that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\beta}}=\\begin{matrix}\\frac{11}{5} \\\\ 2\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ "\n", "## Linking the regression analysis with a statistical interpretation\n", "\n", diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index da229351c..0e3896a03 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -429,6 +429,43 @@ We will now couple the discussions of ordinary least squares, Ridge and Lasso re We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\beta$. +!split +===== Yet another Example ===== + +Let us assume we have a data set with outputs/targets given by the vector + +!bt +\[ +\bm{y}=\begin{matrix}4 \\ 2 \\3\end{bmatrix}, +\] +!et +and our inputs as a $3\times 2$ design matrix +!bt +\[ +\bm{X}=\begin{matrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix}, +\] +!et +meaning that we have two features and two unknown parameters $\beta_0$ and $\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression. + +!split +===== The OLS case ===== + +For ordinary least squares (OLS) we know that the optimal solution is + +!bt +\[ +\hat{\bm{\beta}}=\left( \bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\] +!et +Inserting the above values we obtain that + +!bt +\[ +\hat{\bm{\beta}}=\begin{matrix}\frac{11}{5} \\ 2\end{bmatrix}, +\] +!et + + !split ===== Linking the regression analysis with a statistical interpretation =====