update book chapters

2021-10-26 22:58:15 +02:00
parent 6c32c5b5e2
commit 5ee2934f9a
33 changed files with 367 additions and 202 deletions
@@ -1202,7 +1202,7 @@ standard deviation. Most machine learning libraries do this as a default. This m
 the results may differ. 

 The
-"Standadscaler":"https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html"
+"Standardscaler":"https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html"
 function in _Scikit-Learn_ does this for us.  For the data sets we
 have been studying in our various examples, the data are in many cases
 already scaled and there is no need to scale them. You as a user of different machine learning algorithms, should always perform  a
@@ -1256,12 +1256,16 @@ The cost/loss function  for regression is
 C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
 \]
 !et
-Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.

-What we have done is to single out the $\beta_0$ term in the definition of the mean squared error (MSE).
-The design matrix
-$X$ does in this case not contain any intercept column.
-When we take the derivative with respect to $\beta_0$, we want the derivative to obey
+Recall also that we use the squared value. This expression can lead to an
+increased penalty for higher differences between predicted and
+output/target values.
+
+What we have done is to single out the $\beta_0$ term in the
+definition of the mean squared error (MSE).  The design matrix $X$
+does in this case not contain any intercept column.  When we take the
+derivative with respect to $\beta_0$, we want the derivative to obey
+
 !bt
 \[
 \frac{\partial C}{\partial \beta_j} = 0,
@@ -1282,7 +1286,7 @@ Multiplying away the constant $2/n$, we obtain
 \]
 !et

-Let us special first to the case where we have only two parameters $\beta_0$ and $\beta_1$.
+Let us specialize first to the case where we have only two parameters $\beta_0$ and $\beta_1$.
 Our result for $\beta_0$ simplifies then to
 !bt
 \[
@@ -1298,10 +1302,10 @@ We obtain then
 If we define
 !bt
 \[
-\mu_1=\frac{1}{n}\sum_{i=0}^{n-1} (X_{i1},
+\mu_{\bm{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
 \]
 !et
-and if we define the mean value of the outputs as
+and the mean value of the outputs as
 !bt
 \[
 \mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
@@ -1310,16 +1314,30 @@ and if we define the mean value of the outputs as
 we have
 !bt
 \[
-\beta_0 = \mu_y - \beta_1\mu_{1}.
+\beta_0 = \mu_y - \beta_1\mu_{\bm{x}_1}.
 \]
 !et
-In the general case withmore parameters than $\beta_0$ and $\beta_1$, we have
+In the general case with more parameters than $\beta_0$ and $\beta_1$, we have
 !bt
 \[
 \beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
 \]
 !et

+We can rewrite the latter equation as
+!bt
+\[
+\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\bm{x}_j}\beta_j,
+\]
+!et
+where we have defined
+!bt
+\[
+\mu_{\bm{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
+\]
+!et
+the mean value for all elements of the column vector $\bm{x}_j$.
+


 Replacing $y_i$ with $y_i - y_i - \overline{\bm{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)
@@ -1584,7 +1584,7 @@
    "the results may differ. \n",
    "\n",
    "The\n",
-    "[Standadscaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n",
+    "[Standardscaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n",
    "function in **Scikit-Learn** does this for us.  For the data sets we\n",
    "have been studying in our various examples, the data are in many cases\n",
    "already scaled and there is no need to scale them. You as a user of different machine learning algorithms, should always perform  a\n",
@@ -1659,12 +1659,14 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.\n",
+    "Recall also that we use the squared value. This expression can lead to an\n",
+    "increased penalty for higher differences between predicted and\n",
+    "output/target values.\n",
    "\n",
-    "What we have done is to single out the $\\beta_0$ term in the definition of the mean squared error (MSE).\n",
-    "The design matrix\n",
-    "$X$ does in this case not contain any intercept column.\n",
-    "When we take the derivative with respect to $\\beta_0$, we want the derivative to obey"
+    "What we have done is to single out the $\\beta_0$ term in the\n",
+    "definition of the mean squared error (MSE).  The design matrix $X$\n",
+    "does in this case not contain any intercept column.  When we take the\n",
+    "derivative with respect to $\\beta_0$, we want the derivative to obey"
   ]
  },
  {
@@ -1712,7 +1714,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "Let us special first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
+    "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
    "Our result for $\\beta_0$ simplifies then to"
   ]
  },
@@ -1753,7 +1755,7 @@
   "metadata": {},
   "source": [
    "$$\n",
-    "\\mu_1=\\frac{1}{n}\\sum_{i=0}^{n-1} (X_{i1},\n",
+    "\\mu_{\\boldsymbol{x}_1}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1},\n",
    "$$"
   ]
  },
@@ -1761,7 +1763,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "and if we define the mean value of the outputs as"
+    "and the mean value of the outputs as"
   ]
  },
  {
@@ -1785,7 +1787,7 @@
   "metadata": {},
   "source": [
    "$$\n",
-    "\\beta_0 = \\mu_y - \\beta_1\\mu_{1}.\n",
+    "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n",
    "$$"
   ]
  },
@@ -1793,7 +1795,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "In the general case withmore parameters than $\\beta_0$ and $\\beta_1$, we have"
+    "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have"
   ]
  },
  {
@@ -1809,6 +1811,42 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
+    "We can rewrite the latter equation as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where we have defined"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mu_{\\boldsymbol{x}_j}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{ij},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "the mean value for all elements of the column vector $\\boldsymbol{x}_j$.\n",
+    "\n",
+    "\n",
+    "\n",
    "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)"
   ]
  },
@@ -2868,8 +2906,8 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "5\n",
-    "8\n",
+    "6\n",
+    "0\n",
    " \n",
    "<\n",
    "<\n",
@@ -645,10 +645,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.135976 sec
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.138685 sec
 Jackknife Statistics :
 original           bias      std. error
-  99.655         99.645        0.148675
+ 99.8911        99.8811        0.149732
 </pre></div>
 </div>
 </div>
@@ -867,7 +867,7 @@ theorem.</p>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Bootstrap Statistics :
 original           bias      std. error
- 100.098   15.248        100.098        0.153966
+  99.751  15.2137        99.7503        0.153129
 </pre></div>
 </div>
 </div>
@@ -1092,19 +1092,22 @@ Error: 0.06844519414009442
 Bias^2: 0.06453579006728317
 Var: 0.003909404072811237
 0.06844519414009442 &gt;= 0.06453579006728317 + 0.003909404072811237 = 0.06844519414009441
-Polynomial degree: 5
+Polynomial degree:
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 5
 Error: 0.05227921801205707
 Bias^2: 0.048187277304303125
 Var: 0.004091940707753964
 0.05227921801205707 &gt;= 0.048187277304303125 + 0.004091940707753964 = 0.05227921801205709
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 6
+Polynomial degree: 6
 Error: 0.03781367141738898
 Bias^2: 0.03365768507152761
 Var: 0.004155986345861379
 0.03781367141738898 &gt;= 0.03365768507152761 + 0.004155986345861379 = 0.03781367141738899
-Polynomial degree: 7
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 7
 Error: 0.027609773491022498
 Bias^2: 0.02299949826036597
 Var: 0.004610275230656537
@@ -1119,14 +1122,14 @@ Error: 0.026605727637189085
 Bias^2: 0.010018312644140933
 Var: 0.016587414993048166
 0.026605727637189085 &gt;= 0.010018312644140933 + 0.016587414993048166 = 0.0266057276371891
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 10
+Polynomial degree: 10
 Error: 0.021592704588043153
 Bias^2: 0.010516485576652981
 Var: 0.011076219011390184
 0.021592704588043153 &gt;= 0.010516485576652981 + 0.011076219011390184 = 0.021592704588043167
-Polynomial degree: 11
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 11
 Error: 0.07160048164228314
 Bias^2: 0.01443680008897583
 Var: 0.0571636815533073
@@ -1143,7 +1146,7 @@ Var: 0.20867052175003387
 0.22842468702166951 &gt;= 0.01975416527163567 + 0.20867052175003387 = 0.22842468702166954
 </pre></div>
 </div>
-<img alt="_images/chapter3_62_4.png" src="_images/chapter3_62_4.png" />
+<img alt="_images/chapter3_62_5.png" src="_images/chapter3_62_5.png" />
 </div>
 </div>
 <p>The bias-variance tradeoff summarizes the fundamental tension in
@@ -1378,12 +1381,12 @@ Mean squared error on test data: 8.14812206
 Degree of polynomial:   7
 Mean squared error on training data: 0.47075725
 Mean squared error on test data: 2.00607783
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:   8
+Degree of polynomial:   8
 Mean squared error on training data: 0.04912436
 Mean squared error on test data: 0.21596432
-Degree of polynomial:   9
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:   9
 Mean squared error on training data: 0.02522069
 Mean squared error on test data: 0.08576932
 Degree of polynomial:  10
@@ -1392,56 +1395,54 @@ Mean squared error on test data: 1.20015436
 Degree of polynomial:  11
 Mean squared error on training data: 0.01640891
 Mean squared error on test data: 1.35533774
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  12
+Degree of polynomial:  12
 Mean squared error on training data: 0.00813803
 Mean squared error on test data: 0.17446471
-Degree of polynomial:  13
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  13
 Mean squared error on training data: 0.00759119
 Mean squared error on test data: 1.08131001
 Degree of polynomial:  14
 Mean squared error on training data: 0.00472199
 Mean squared error on test data: 0.81333802
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  15
+Degree of polynomial:  15
 Mean squared error on training data: 0.00410478
 Mean squared error on test data: 92.09160813
-Degree of polynomial:  16
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  16
 Mean squared error on training data: 0.00315593
 Mean squared error on test data: 234.40530431
 Degree of polynomial:  17
 Mean squared error on training data: 0.00242999
 Mean squared error on test data: 1270.94936405
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  18
+Degree of polynomial:  18
 Mean squared error on training data: 0.00228741
 Mean squared error on test data: 108.11945731
 Degree of polynomial:  19
 Mean squared error on training data: 0.00156372
 Mean squared error on test data: 1376.61081005
-Degree of polynomial:  20
-Mean squared error on training data: 0.00137945
-Mean squared error on test data: 1931.97211078
 </pre></div>
 </div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  21
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  20
+Mean squared error on training data: 0.00137945
+Mean squared error on test data: 1931.97211078
+Degree of polynomial:  21
 Mean squared error on training data: 0.00118678
 Mean squared error on test data: 14496.70992192
 Degree of polynomial:  22
 Mean squared error on training data: 0.00092686
 Mean squared error on test data: 873.95463048
-Degree of polynomial:  23
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  23
 Mean squared error on training data: 0.00085890
 Mean squared error on test data: 5535.20053452
 Degree of polynomial:  24
 Mean squared error on training data: 0.00084714
 Mean squared error on test data: 1289.22422186
-</pre></div>
-</div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial:  25
+Degree of polynomial:  25
 Mean squared error on training data: 0.00079022
 Mean squared error on test data: 136582.88824397
 Degree of polynomial:  26
@@ -1466,7 +1467,7 @@ Mean squared error on test data: 3204.07047448
  plt.plot(polynomial, np.log10(testerror), label=&#39;Test Error&#39;)
 </pre></div>
 </div>
-<img alt="_images/chapter3_65_10.png" src="_images/chapter3_65_10.png" />
+<img alt="_images/chapter3_65_9.png" src="_images/chapter3_65_9.png" />
 </div>
 </div>
 </div>
@@ -1729,7 +1730,7 @@ column from the corresponding column and dividing the column with its
 standard deviation. Most machine learning libraries do this as a default. This means that if you compare your code with the results from a given library,
 the results may differ.</p>
 <p>The
-<a class="reference external" href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">Standadscaler</a>
+<a class="reference external" href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">Standardscaler</a>
 function in <strong>Scikit-Learn</strong> does this for us.  For the data sets we
 have been studying in our various examples, the data are in many cases
 already scaled and there is no need to scale them. You as a user of different machine learning algorithms, should always perform  a
@@ -1781,11 +1782,13 @@ simplicity, we will focus on  ordinary regression, as done in the above example.
 \[
 C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
 \]</div>
-<p>Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.</p>
-<p>What we have done is to single out the <span class="math notranslate nohighlight">\(\beta_0\)</span> term in the definition of the mean squared error (MSE).
-The design matrix
-<span class="math notranslate nohighlight">\(X\)</span> does in this case not contain any intercept column.
-When we take the derivative with respect to <span class="math notranslate nohighlight">\(\beta_0\)</span>, we want the derivative to obey</p>
+<p>Recall also that we use the squared value. This expression can lead to an
+increased penalty for higher differences between predicted and
+output/target values.</p>
+<p>What we have done is to single out the <span class="math notranslate nohighlight">\(\beta_0\)</span> term in the
+definition of the mean squared error (MSE).  The design matrix <span class="math notranslate nohighlight">\(X\)</span>
+does in this case not contain any intercept column.  When we take the
+derivative with respect to <span class="math notranslate nohighlight">\(\beta_0\)</span>, we want the derivative to obey</p>
 <div class="math notranslate nohighlight">
 \[
 \frac{\partial C}{\partial \beta_j} = 0,
@@ -1800,7 +1803,7 @@ When we take the derivative with respect to <span class="math notranslate nohigh
 \[
 \sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
 \]</div>
-<p>Let us special first to the case where we have only two parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>.
+<p>Let us specialize first to the case where we have only two parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>.
 Our result for <span class="math notranslate nohighlight">\(\beta_0\)</span> simplifies then to</p>
 <div class="math notranslate nohighlight">
 \[
@@ -1814,9 +1817,9 @@ n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
 <p>If we define</p>
 <div class="math notranslate nohighlight">
 \[
-\mu_1=\frac{1}{n}\sum_{i=0}^{n-1} (X_{i1},
+\mu_{\boldsymbol{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
 \]</div>
-<p>and if we define the mean value of the outputs as</p>
+<p>and the mean value of the outputs as</p>
 <div class="math notranslate nohighlight">
 \[
 \mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
@@ -1824,13 +1827,24 @@ n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
 <p>we have</p>
 <div class="math notranslate nohighlight">
 \[
-\beta_0 = \mu_y - \beta_1\mu_{1}.
+\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}.
 \]</div>
-<p>In the general case withmore parameters than <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>, we have</p>
+<p>In the general case with more parameters than <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span>, we have</p>
 <div class="math notranslate nohighlight">
 \[
 \beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
 \]</div>
+<p>We can rewrite the latter equation as</p>
+<div class="math notranslate nohighlight">
+\[
+\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j,
+\]</div>
+<p>where we have defined</p>
+<div class="math notranslate nohighlight">
+\[
+\mu_{\boldsymbol{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
+\]</div>
+<p>the mean value for all elements of the column vector <span class="math notranslate nohighlight">\(\boldsymbol{x}_j\)</span>.</p>
 <p>Replacing <span class="math notranslate nohighlight">\(y_i\)</span> with <span class="math notranslate nohighlight">\(y_i - y_i - \overline{\boldsymbol{y}}\)</span> and centering also our design matrix results in a cost function (in vector-matrix disguise)</p>
 <div class="math notranslate nohighlight">
 \[
@@ -1960,7 +1974,7 @@ MSE with Sklearn intercept
 0.004113634617443135
 </pre></div>
 </div>
-<img alt="_images/chapter3_103_1.png" src="_images/chapter3_103_1.png" />
+<img alt="_images/chapter3_107_1.png" src="_images/chapter3_107_1.png" />
 </div>
 </div>
 <p>The intercept is the value of our output/target variable
@@ -2155,7 +2169,7 @@ MSE values for Scikit-Learn Ridge implementation
 0.26409315307910036
 </pre></div>
 </div>
-<img alt="_images/chapter3_111_1.png" src="_images/chapter3_111_1.png" />
+<img alt="_images/chapter3_115_1.png" src="_images/chapter3_115_1.png" />
 </div>
 </div>
 <p>The results here agree when we force <strong>Scikit-Learn</strong>’s Ridge function to include the first column in our design matrix.
@@ -2358,7 +2372,7 @@ MSE values for Scikit-Learn Ridge implementation
 0.002381316302584886
 </pre></div>
 </div>
-<img alt="_images/chapter3_113_1.png" src="_images/chapter3_113_1.png" />
+<img alt="_images/chapter3_117_1.png" src="_images/chapter3_117_1.png" />
 </div>
 </div>
 <p>We see here, when compared to the code which includes explicitely the
@@ -2576,7 +2590,7 @@ linear system as an equation would reduce this down to
  cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
 </pre></div>
 </div>
-<img alt="_images/chapter3_144_1.png" src="_images/chapter3_144_1.png" />
+<img alt="_images/chapter3_148_1.png" src="_images/chapter3_148_1.png" />
 </div>
 </div>
 <p>It is interesting to note that OLS
@@ -2720,7 +2734,7 @@ with the form utilized in linear regression, viz.</p>
  cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
 </pre></div>
 </div>
-<img alt="_images/chapter3_162_1.png" src="_images/chapter3_162_1.png" />
+<img alt="_images/chapter3_166_1.png" src="_images/chapter3_166_1.png" />
 </div>
 </div>
 <p>The results agree perfectly with our previous discussion where we used our own code.</p>
@@ -2729,8 +2743,8 @@ regression. In ridge regression we include a <strong>regularizer</strong>. This
 involves a new cost function which leads to a new estimate for the
 weights <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. This results in a penalized regression problem. The
 cost function is given by</p>
-<p>5
-8</p>
+<p>6
+0</p>
 <p>&lt;
 &lt;
 &lt;
@@ -2768,7 +2782,7 @@ K</p>
  cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
 </pre></div>
 </div>
-<img alt="_images/chapter3_165_1.png" src="_images/chapter3_165_1.png" />
+<img alt="_images/chapter3_169_1.png" src="_images/chapter3_169_1.png" />
 </div>
 </div>
 <p>In the <strong>Least Absolute Shrinkage and Selection Operator</strong> (LASSO)-method we get a third cost function.</p>
@@ -2803,7 +2817,7 @@ K</p>
  cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
 </pre></div>
 </div>
-<img alt="_images/chapter3_169_1.png" src="_images/chapter3_169_1.png" />
+<img alt="_images/chapter3_173_1.png" src="_images/chapter3_173_1.png" />
 </div>
 </div>
 <p>It is quite striking how LASSO breaks the symmetry of the coupling
@@ -2861,40 +2875,40 @@ constant as opposed to ridge and OLS. We get a sparse solution with
 10%|█         | 1/10 [00:00&lt;00:04,  2.02it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 20%|██        | 2/10 [00:00&lt;00:03,  2.25it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 20%|██        | 2/10 [00:00&lt;00:03,  2.36it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 30%|███       | 3/10 [00:00&lt;00:02,  2.82it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 30%|███       | 3/10 [00:00&lt;00:02,  2.91it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 40%|████      | 4/10 [00:01&lt;00:01,  3.33it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 40%|████      | 4/10 [00:01&lt;00:01,  3.60it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 50%|█████     | 5/10 [00:01&lt;00:01,  3.94it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 50%|█████     | 5/10 [00:01&lt;00:01,  4.36it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 60%|██████    | 6/10 [00:01&lt;00:00,  4.61it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 60%|██████    | 6/10 [00:01&lt;00:00,  4.87it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 70%|███████   | 7/10 [00:01&lt;00:00,  5.17it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 70%|███████   | 7/10 [00:01&lt;00:00,  5.64it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 80%|████████  | 8/10 [00:01&lt;00:00,  5.73it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 80%|████████  | 8/10 [00:01&lt;00:00,  6.31it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 90%|█████████ | 9/10 [00:01&lt;00:00,  6.46it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 90%|█████████ | 9/10 [00:01&lt;00:00,  6.88it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|██████████| 10/10 [00:01&lt;00:00,  6.87it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|██████████| 10/10 [00:01&lt;00:00,  7.10it/s]
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|██████████| 10/10 [00:01&lt;00:00,  5.22it/s]
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|██████████| 10/10 [00:01&lt;00:00,  5.64it/s]
 </pre></div>
 </div>
 <div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
 </pre></div>
 </div>
-<img alt="_images/chapter3_171_13.png" src="_images/chapter3_171_13.png" />
+<img alt="_images/chapter3_175_13.png" src="_images/chapter3_175_13.png" />
 </div>
 </div>
 <p>We see that LASSO reaches a good solution for low
@@ -2943,7 +2957,7 @@ testing set that is close to the accuracy of the training set.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<img alt="_images/chapter3_173_0.png" src="_images/chapter3_173_0.png" />
+<img alt="_images/chapter3_177_0.png" src="_images/chapter3_177_0.png" />
 </div>
 </div>
 <p>From the above figure we can see that LASSO with <span class="math notranslate nohighlight">\(\lambda = 10^{-2}\)</span>
@@ -3035,7 +3049,7 @@ which polynomial fits the data best.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<img alt="_images/chapter3_177_0.png" src="_images/chapter3_177_0.png" />
+<img alt="_images/chapter3_181_0.png" src="_images/chapter3_181_0.png" />
 </div>
 </div>
 <div class="section" id="exercise-ordinary-least-square-ols-on-the-franke-function">
@@ -1163,7 +1163,7 @@ standard deviation. Most machine learning libraries do this as a default. This m
 the results may differ. 

 The
-[Standadscaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)
+[Standardscaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)
 function in **Scikit-Learn** does this for us.  For the data sets we
 have been studying in our various examples, the data are in many cases
 already scaled and there is no need to scale them. You as a user of different machine learning algorithms, should always perform  a
@@ -1213,12 +1213,14 @@ $$
 C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
 $$

-Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.
+Recall also that we use the squared value. This expression can lead to an
+increased penalty for higher differences between predicted and
+output/target values.

-What we have done is to single out the $\beta_0$ term in the definition of the mean squared error (MSE).
-The design matrix
-$X$ does in this case not contain any intercept column.
-When we take the derivative with respect to $\beta_0$, we want the derivative to obey
+What we have done is to single out the $\beta_0$ term in the
+definition of the mean squared error (MSE).  The design matrix $X$
+does in this case not contain any intercept column.  When we take the
+derivative with respect to $\beta_0$, we want the derivative to obey

 $$
 \frac{\partial C}{\partial \beta_j} = 0,
@@ -1236,7 +1238,7 @@ $$
 \sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
 $$

-Let us special first to the case where we have only two parameters $\beta_0$ and $\beta_1$.
+Let us specialize first to the case where we have only two parameters $\beta_0$ and $\beta_1$.
 Our result for $\beta_0$ simplifies then to

 $$
@@ -1252,10 +1254,10 @@ $$
 If we define

 $$
-\mu_1=\frac{1}{n}\sum_{i=0}^{n-1} (X_{i1},
+\mu_{\boldsymbol{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
 $$

-and if we define the mean value of the outputs as
+and the mean value of the outputs as

 $$
 \mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
@@ -1264,15 +1266,31 @@ $$
 we have

 $$
-\beta_0 = \mu_y - \beta_1\mu_{1}.
+\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}.
 $$

-In the general case withmore parameters than $\beta_0$ and $\beta_1$, we have
+In the general case with more parameters than $\beta_0$ and $\beta_1$, we have

 $$
 \beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
 $$

+We can rewrite the latter equation as
+
+$$
+\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j,
+$$
+
+where we have defined
+
+$$
+\mu_{\boldsymbol{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
+$$
+
+the mean value for all elements of the column vector $\boldsymbol{x}_j$.
+
+
+
 Replacing $y_i$ with $y_i - y_i - \overline{\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)

 $$
@@ -1917,8 +1935,8 @@ involves a new cost function which leads to a new estimate for the
 weights $\boldsymbol{\beta}$. This results in a penalized regression problem. The
 cost function is given by

-5
-8
+6
+0
 
 <
 <
@@ -1584,7 +1584,7 @@
    "the results may differ. \n",
    "\n",
    "The\n",
-    "[Standadscaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n",
+    "[Standardscaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n",
    "function in **Scikit-Learn** does this for us.  For the data sets we\n",
    "have been studying in our various examples, the data are in many cases\n",
    "already scaled and there is no need to scale them. You as a user of different machine learning algorithms, should always perform  a\n",
@@ -1659,12 +1659,14 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.\n",
+    "Recall also that we use the squared value. This expression can lead to an\n",
+    "increased penalty for higher differences between predicted and\n",
+    "output/target values.\n",
    "\n",
-    "What we have done is to single out the $\\beta_0$ term in the definition of the mean squared error (MSE).\n",
-    "The design matrix\n",
-    "$X$ does in this case not contain any intercept column.\n",
-    "When we take the derivative with respect to $\\beta_0$, we want the derivative to obey"
+    "What we have done is to single out the $\\beta_0$ term in the\n",
+    "definition of the mean squared error (MSE).  The design matrix $X$\n",
+    "does in this case not contain any intercept column.  When we take the\n",
+    "derivative with respect to $\\beta_0$, we want the derivative to obey"
   ]
  },
  {
@@ -1712,7 +1714,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "Let us special first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
+    "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
    "Our result for $\\beta_0$ simplifies then to"
   ]
  },
@@ -1753,7 +1755,7 @@
   "metadata": {},
   "source": [
    "$$\n",
-    "\\mu_1=\\frac{1}{n}\\sum_{i=0}^{n-1} (X_{i1},\n",
+    "\\mu_{\\boldsymbol{x}_1}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1},\n",
    "$$"
   ]
  },
@@ -1761,7 +1763,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "and if we define the mean value of the outputs as"
+    "and the mean value of the outputs as"
   ]
  },
  {
@@ -1785,7 +1787,7 @@
   "metadata": {},
   "source": [
    "$$\n",
-    "\\beta_0 = \\mu_y - \\beta_1\\mu_{1}.\n",
+    "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n",
    "$$"
   ]
  },
@@ -1793,7 +1795,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "In the general case withmore parameters than $\\beta_0$ and $\\beta_1$, we have"
+    "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have"
   ]
  },
  {
@@ -1809,6 +1811,42 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
+    "We can rewrite the latter equation as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where we have defined"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mu_{\\boldsymbol{x}_j}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{ij},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "the mean value for all elements of the column vector $\\boldsymbol{x}_j$.\n",
+    "\n",
+    "\n",
+    "\n",
    "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)"
   ]
  },
@@ -2868,8 +2906,8 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "5\n",
-    "8\n",
+    "6\n",
+    "0\n",
    " \n",
    "<\n",
    "<\n",