lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

correcting typos

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-08-28 09:19:48 +02:00
parent 9ac005c004
commit cf0945f94f
11 changed files with 462 additions and 814 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/._week35-bs001.html
+2 -2
View File
@@ -382,8 +382,8 @@ MathJax.Hub.Config({
<ol>
<li> See lecture notes for week 35 at <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_self"><tt>https://compphysics.github.io/MachineLearning/doc/web/course.html</tt></a></li>
<li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</li>
<li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</li>
<li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)</li>
<li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4 (on relevance for the discussion of linear regression).</li>
</ol>
<p>
<!-- navigation buttons at the bottom of the page -->
doc/pub/week35/html/._week35-bs006.html
+2 -3
View File
@@ -388,14 +388,13 @@ $$
<p>We note also that since our design matrix is defined as \( \boldsymbol{X}\in
{\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in
{\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \),
in our case \( p=5 \) meaning that we end up with inverting a small
{\mathbb{R}}^{p\times p} \). In most cases we have that \( p \ll n \). In our example case below we have \( p=5 \) meaning. We end up with inverting a small
\( 5\times 5 \) matrix. This is a rather common situation, in many cases we end up with low-dimensional
matrices to invert. The methods discussed here and for many other
supervised learning algorithms like classification with logistic
regression or support vector machines, exhibit dimensionalities which
allow for the usage of direct linear algebra methods such as <b>LU</b> decomposition or <b>Singular Value Decomposition</b> (SVD) for finding the inverse of the matrix
\( \boldsymbol{X}^T\boldsymbol{X} \).
\( \boldsymbol{X}^T\boldsymbol{X} \). This is discussed on Thursday this week.
</p>
</div>
</div>
doc/pub/week35/html/._week35-bs013.html
+1 -1
View File
@@ -383,7 +383,7 @@ $$
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
</p>
<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
$$
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
$$
doc/pub/week35/html/._week35-bs018.html
+1
View File
@@ -378,6 +378,7 @@ MathJax.Hub.Config({
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #408080; font-style: italic"># matrix inversion to find beta</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
beta <span style="color: #666666">=</span> (np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X) <span style="color: #666666">@</span> X<span style="color: #666666">.</span>T ) <span style="color: #666666">@</span> y
<span style="color: #408080; font-style: italic"># and then make the prediction</span>
ytilde <span style="color: #666666">=</span> X <span style="color: #666666">@</span> beta
doc/pub/week35/html/week35-reveal.html
+6 -6
View File
@@ -211,8 +211,8 @@ MathJax.Hub.Config({
<ol>
<p><li> See lecture notes for week 35 at <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/web/course.html</tt></a></li>
<p><li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</li>
<p><li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</li>
<p><li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)</li>
<p><li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4 (on relevance for the discussion of linear regression).</li>
</ol>
</section>
@@ -409,14 +409,13 @@ $$
<p>We note also that since our design matrix is defined as \( \boldsymbol{X}\in
{\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in
{\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \),
in our case \( p=5 \) meaning that we end up with inverting a small
{\mathbb{R}}^{p\times p} \). In most cases we have that \( p \ll n \). In our example case below we have \( p=5 \) meaning. We end up with inverting a small
\( 5\times 5 \) matrix. This is a rather common situation, in many cases we end up with low-dimensional
matrices to invert. The methods discussed here and for many other
supervised learning algorithms like classification with logistic
regression or support vector machines, exhibit dimensionalities which
allow for the usage of direct linear algebra methods such as <b>LU</b> decomposition or <b>Singular Value Decomposition</b> (SVD) for finding the inverse of the matrix
\( \boldsymbol{X}^T\boldsymbol{X} \).
\( \boldsymbol{X}^T\boldsymbol{X} \). This is discussed on Thursday this week.
</p>
</div>
@@ -666,7 +665,7 @@ $$
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
</p>
<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
<p>&nbsp;<br>
$$
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
@@ -829,6 +828,7 @@ $$
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="font-size: 80%; line-height: 125%;"><span style="color: #228B22"># matrix inversion to find beta</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
<span style="color: #228B22"># and then make the prediction</span>
ytilde = X @ beta
doc/pub/week35/html/week35-solarized.html
+6 -6
View File
@@ -332,8 +332,8 @@ MathJax.Hub.Config({
<ol>
<li> See lecture notes for week 35 at <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/web/course.html</tt></a></li>
<li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</li>
<li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</li>
<li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)</li>
<li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4 (on relevance for the discussion of linear regression).</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week </h2>
@@ -494,14 +494,13 @@ $$
<p>We note also that since our design matrix is defined as \( \boldsymbol{X}\in
{\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in
{\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \),
in our case \( p=5 \) meaning that we end up with inverting a small
{\mathbb{R}}^{p\times p} \). In most cases we have that \( p \ll n \). In our example case below we have \( p=5 \) meaning. We end up with inverting a small
\( 5\times 5 \) matrix. This is a rather common situation, in many cases we end up with low-dimensional
matrices to invert. The methods discussed here and for many other
supervised learning algorithms like classification with logistic
regression or support vector machines, exhibit dimensionalities which
allow for the usage of direct linear algebra methods such as <b>LU</b> decomposition or <b>Singular Value Decomposition</b> (SVD) for finding the inverse of the matrix
\( \boldsymbol{X}^T\boldsymbol{X} \).
\( \boldsymbol{X}^T\boldsymbol{X} \). This is discussed on Thursday this week.
</p>
</div>
@@ -708,7 +707,7 @@ $$
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
</p>
<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
$$
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
$$
@@ -839,6 +838,7 @@ $$
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #228B22"># matrix inversion to find beta</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
<span style="color: #228B22"># and then make the prediction</span>
ytilde = X @ beta
doc/pub/week35/html/week35.html
+6 -6
View File
@@ -409,8 +409,8 @@ MathJax.Hub.Config({
<ol>
<li> See lecture notes for week 35 at <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/web/course.html</tt></a></li>
<li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</li>
<li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</li>
<li> Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)</li>
<li> Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4 (on relevance for the discussion of linear regression).</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week </h2>
@@ -571,14 +571,13 @@ $$
<p>We note also that since our design matrix is defined as \( \boldsymbol{X}\in
{\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in
{\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \),
in our case \( p=5 \) meaning that we end up with inverting a small
{\mathbb{R}}^{p\times p} \). In most cases we have that \( p \ll n \). In our example case below we have \( p=5 \) meaning. We end up with inverting a small
\( 5\times 5 \) matrix. This is a rather common situation, in many cases we end up with low-dimensional
matrices to invert. The methods discussed here and for many other
supervised learning algorithms like classification with logistic
regression or support vector machines, exhibit dimensionalities which
allow for the usage of direct linear algebra methods such as <b>LU</b> decomposition or <b>Singular Value Decomposition</b> (SVD) for finding the inverse of the matrix
\( \boldsymbol{X}^T\boldsymbol{X} \).
\( \boldsymbol{X}^T\boldsymbol{X} \). This is discussed on Thursday this week.
</p>
</div>
@@ -785,7 +784,7 @@ $$
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
</p>
<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
$$
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
$$
@@ -916,6 +915,7 @@ $$
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #408080; font-style: italic"># matrix inversion to find beta</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
beta <span style="color: #666666">=</span> (np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X) <span style="color: #666666">@</span> X<span style="color: #666666">.</span>T ) <span style="color: #666666">@</span> y
<span style="color: #408080; font-style: italic"># and then make the prediction</span>
ytilde <span style="color: #666666">=</span> X <span style="color: #666666">@</span> beta
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+432 -432
View File
File diff suppressed because it is too large Load Diff
doc/src/week35/exercisesweek35.ipynb
-352
View File
@@ -1,352 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "28dad322",
"metadata": {},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek35.do.txt -->\n",
"<!-- dom:TITLE: Exercises week 35 -->"
]
},
{
"cell_type": "markdown",
"id": "9c719aee",
"metadata": {},
"source": [
"# Exercises week 35\n",
"**August 28-September 1, 2023**\n",
"\n",
"Date: **Deadline, Friday September 1 at midnight**"
]
},
{
"cell_type": "markdown",
"id": "cb1c110e",
"metadata": {},
"source": [
"## Exercise 1: Analytical exercises\n",
"\n",
"In this exercise we derive the expressions for various derivatives of\n",
"products of vectors and matrices. Such derivatives are central to the\n",
"optimization of various cost functions. Although we will often use\n",
"automatic differentiation in actual calculations, to be able to have\n",
"analytical expressions is extremely helpful in case we have simpler\n",
"derivatives as well as when we analyze various properties (like second\n",
"derivatives) of the chosen cost functions. Vectors are always written\n",
"as boldfaced lower case letters and matrices as upper case boldfaced\n",
"letters. You will find useful the notes from week 35 on derivatives of vectors and matrices.\n",
"\n",
"Show that"
]
},
{
"cell_type": "markdown",
"id": "7e7a925d",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial (\\boldsymbol{b}^T\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{b},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "1e9b2076",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"id": "451ff3bd",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{A}\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{a}^T(\\boldsymbol{A}+\\boldsymbol{A}^T),\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "d44390bf",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"id": "445324c2",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)}{\\partial \\boldsymbol{s}} = -2\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\boldsymbol{A},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "e7405149",
"metadata": {},
"source": [
"and finally find the second derivative of this function with respect to the vector $\\boldsymbol{s}$. If we replace the vector $\\boldsymbol{s}$ with the unknown parameters $\\boldsymbol{\\beta}$ used to define the ordinary least squares method, we end up with the equations that determine these parameters. The matrix $\\boldsymbol{A}$ is then the design matrix $\\boldsymbol{X}$ and $\\boldsymbol{x}$ here has to be replaced with the outputs $\\boldsymbol{y}$.\n",
"\n",
"The second derivative of the mean squared error is then proportional to the so-called Hessian matrix $\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}$.\n",
"\n",
"**Hint**: In these exercises it is always useful to write out with summation indices the various quantities.\n",
"As an example, consider the function"
]
},
{
"cell_type": "markdown",
"id": "3b7eb2f9",
"metadata": {},
"source": [
"$$\n",
"f(\\boldsymbol{x}) =\\boldsymbol{A}\\boldsymbol{x},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "52cc515a",
"metadata": {},
"source": [
"which reads for a specific component $f_i$ (we define the matrix $\\boldsymbol{A}$ to have dimension $n\\times n$ and the vector $\\boldsymbol{x}$ to have length $n$)"
]
},
{
"cell_type": "markdown",
"id": "a76d3703",
"metadata": {},
"source": [
"$$\n",
"f_i =\\sum_{j=0}^{n-1}a_{ij}x_j,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "79edf0b7",
"metadata": {},
"source": [
"which leads to"
]
},
{
"cell_type": "markdown",
"id": "14dd340b",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f_i}{\\partial x_j}= a_{ij},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "839010a1",
"metadata": {},
"source": [
"and written out in terms of the vector $\\boldsymbol{x}$ we have"
]
},
{
"cell_type": "markdown",
"id": "f53fa6eb",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f(\\boldsymbol{x})}{\\partial \\boldsymbol{x}}= \\boldsymbol{A}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "fc259c9f",
"metadata": {},
"source": [
"## Exercise 2: making your own data and exploring scikit-learn\n",
"\n",
"We will generate our own dataset for a function $y(x)$ where $x \\in\n",
"[0,1]$ and defined by random numbers computed with the uniform\n",
"distribution. The function $y$ is a quadratic polynomial in $x$ with\n",
"added stochastic noise according to the normal distribution $\\cal\n",
"{N}(0,1)$. The following simple Python instructions define our $x$\n",
"and $y$ values (with 100 data points)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "a4ad5843",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"x = np.random.rand(100,1)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100,1)"
]
},
{
"cell_type": "markdown",
"id": "f2162d9e",
"metadata": {},
"source": [
"1. Write your own code (following the examples under the [regression notes](https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter1.html)) for computing the parametrization of the data set fitting a second-order polynomial. \n",
"\n",
"2. Use thereafter **scikit-learn** (see again the examples in the regression slides) and compare with your own code. Note here that **scikit-learn** does not include, by default, the intercept. See the discussions on scaling your data in the slides for this week. This type of problems appear in particular if we fit a polynomial with an intercept. \n",
"\n",
"3. Using scikit-learn, compute also the mean squared error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as"
]
},
{
"cell_type": "markdown",
"id": "3e68b705",
"metadata": {},
"source": [
"$$\n",
"MSE(\\boldsymbol{y},\\boldsymbol{\\tilde{y}}) = \\frac{1}{n}\n",
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "9bcd4cf4",
"metadata": {},
"source": [
"and the $R^2$ score function.\n",
"If $\\tilde{\\boldsymbol{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
]
},
{
"cell_type": "markdown",
"id": "bece94d5",
"metadata": {},
"source": [
"$$\n",
"R^2(\\boldsymbol{y}, \\tilde{\\boldsymbol{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "ec3afe62",
"metadata": {},
"source": [
"where we have defined the mean value of $\\boldsymbol{y}$ as"
]
},
{
"cell_type": "markdown",
"id": "bf20e50c",
"metadata": {},
"source": [
"$$\n",
"\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "ce77b9f1",
"metadata": {},
"source": [
"You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions. \n",
"Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits."
]
},
{
"cell_type": "markdown",
"id": "68933736",
"metadata": {},
"source": [
"## Exercise 3: Split data in test and training data\n",
"\n",
"In this exercise we want you to to compute the MSE for the training\n",
"data and the test data as function of the complexity of a polynomial,\n",
"that is the degree of a given polynomial. \n",
"\n",
"The aim is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf).\n",
"Feel free to read the discussions leading to figure 2.11 of Hastie et al. \n",
"\n",
"Our data is defined by $x\\in [-3,3]$ with a total of for example $n=100$ data points. You should try to vary the number of data points $n$ in your analysis."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e37ee7a9",
"metadata": {},
"outputs": [],
"source": [
"np.random.seed()\n",
"n = 100\n",
"# Make data set.\n",
"x = np.linspace(-3, 3, n).reshape(-1, 1)\n",
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)"
]
},
{
"cell_type": "markdown",
"id": "b096a6a8",
"metadata": {},
"source": [
"where $y$ is the function we want to fit with a given polynomial."
]
},
{
"cell_type": "markdown",
"id": "62e4b8f7",
"metadata": {},
"source": [
"**a)**\n",
"Write a first code which sets up a design matrix $X$ defined by a fifth-order polynomial and split your data set in training and test data."
]
},
{
"cell_type": "markdown",
"id": "3d9241eb",
"metadata": {},
"source": [
"**b)**\n",
"Write thereafter (using either **scikit-learn** or your matrix inversion code using for example **numpy**)\n",
"and perform an ordinary least squares fitting and compute the mean squared error for the training data and the test data. These calculations should apply to a model given by a fifth-order polynomial. If you compare your own code with _scikit_learn_, not that the latter does not include by default the intercept. See the discussions on scaling your data in the slides for this week."
]
},
{
"cell_type": "markdown",
"id": "6e6d0041",
"metadata": {},
"source": [
"**c)**\n",
"Add now a model which allows you to make polynomials up to degree $15$. Perform a standard OLS fitting of the training data and compute the MSE for the training and test data and plot both test and training data MSE as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/src/week35/week35.do.txt
+6 -6
View File
@@ -19,8 +19,8 @@ o Thursday: Ridge and Lasso regression and Singular Value Decomposition
=== Reading recommendations: ===
o See lecture notes for week 35 at URL:"https://compphysics.github.io/MachineLearning/doc/web/course.html"
o Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics
o Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4
o Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)
o Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4 (on relevance for the discussion of linear regression).
@@ -211,14 +211,13 @@ and if the matrix $\bm{X}^T\bm{X}$ is invertible we have the solution
We note also that since our design matrix is defined as $\bm{X}\in
{\mathbb{R}}^{n\times p}$, the product $\bm{X}^T\bm{X} \in
{\mathbb{R}}^{p\times p}$. In the above case we have that $p \ll n$,
in our case $p=5$ meaning that we end up with inverting a small
{\mathbb{R}}^{p\times p}$. In most cases we have that $p \ll n$. In our example case below we have $p=5$ meaning. We end up with inverting a small
$5\times 5$ matrix. This is a rather common situation, in many cases we end up with low-dimensional
matrices to invert. The methods discussed here and for many other
supervised learning algorithms like classification with logistic
regression or support vector machines, exhibit dimensionalities which
allow for the usage of direct linear algebra methods such as _LU_ decomposition or _Singular Value Decomposition_ (SVD) for finding the inverse of the matrix
$\bm{X}^T\bm{X}$.
$\bm{X}^T\bm{X}$. This is discussed on Thursday this week.
!eblock
!bblock
@@ -439,7 +438,7 @@ C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{
We note that the design matrix $\bm{X}$ does not depend on the unknown parameters defined by the vector $\bm{\beta}$.
We are now interested in minimizing the cost function with respect to the unknown parameters $\bm{\beta}$.
The mean squared error is a scalar and if we use the results from the last example, we define a new vector
The mean squared error is a scalar and if we use the results from example three above, we can define a new vector
!bt
\[
\bm{w}=\bm{y}-\bm{X}\bm{\beta},
@@ -584,6 +583,7 @@ $p\times n$ matrix $\bm{X}$.
It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\beta}$. After having defined the matrix $\bm{X}$ and the outputs $\bm{y}$ we have
!bc pycod
# matrix inversion to find beta
import numpy as np
beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
# and then make the prediction
ytilde = X @ beta