correcting typos

doc/LectureNotes/_build/html/week35.html

@@ -1313,7 +1313,7 @@ given function <span class="math notranslate nohighlight">\(f\)</span></p>
 <p>We define the partial derivatives of the various components of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as functions of <span class="math notranslate nohighlight">\(x_i\)</span> in terms of the so-called <a class="reference external" href="https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant">Jacobian matrix</a></p>
 <div class="math notranslate nohighlight">
 \[\begin{split}
-\boldsymbol{J}=\frac{\partial \boldsymbol{y}}{\partial \boldsymbol{x}}=\begin{bmatrix} \frac{\partial y_0}{\partial x_0} &amp; \frac{\partial y_0}{\partial x_1} &amp; \frac{\partial y_0}{\partial x_2} &amp; \dots &amp; \dots &amp; \frac{\partial y_0}{\partial x_{n-1}} \\ \frac{\partial y_0}{\partial x_0} &amp; \frac{\partial y_1}{\partial x_1} &amp; \frac{\partial y_1}{\partial x_2} &amp; \dots &amp; \dots &amp; \frac{\partial y_1}{\partial x_{n-1}} \\
+\boldsymbol{J}=\frac{\partial \boldsymbol{y}}{\partial \boldsymbol{x}}=\begin{bmatrix} \frac{\partial y_0}{\partial x_0} &amp; \frac{\partial y_0}{\partial x_1} &amp; \frac{\partial y_0}{\partial x_2} &amp; \dots &amp; \dots &amp; \frac{\partial y_0}{\partial x_{n-1}} \\ \frac{\partial y_1}{\partial x_0} &amp; \frac{\partial y_1}{\partial x_1} &amp; \frac{\partial y_1}{\partial x_2} &amp; \dots &amp; \dots &amp; \frac{\partial y_1}{\partial x_{n-1}} \\
 \frac{\partial y_2}{\partial x_0} &amp; \frac{\partial y_2}{\partial x_1} &amp; \frac{\partial y_2}{\partial x_2} &amp; \dots &amp; \dots &amp; \frac{\partial y_2}{\partial x_{n-1}} \\
 \dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
 \dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots &amp; \dots \\
@@ -1358,7 +1358,7 @@ multiplications</p>
 \]</div>
 <p>with <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> a vector of length <span class="math notranslate nohighlight">\(m\)</span>, <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> an <span class="math notranslate nohighlight">\(m\times n\)</span> matrix and <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> a vector of length <span class="math notranslate nohighlight">\(n\)</span>. We assume also that <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> does not depend on any of the two vectors.
 In order to find the derivative of <span class="math notranslate nohighlight">\(\alpha\)</span> with respect to the two vectors, we define an intermediate vector <span class="math notranslate nohighlight">\(\boldsymbol{z}\)</span>. We define first
-<span class="math notranslate nohighlight">\(\boldsymbol{z}^T=\boldsymbol{y}^T\boldsymbol{A}\)</span>, a vector of length <span class="math notranslate nohighlight">\(n\)</span>. We have then</p>
+<span class="math notranslate nohighlight">\(\boldsymbol{z}^T=\boldsymbol{y}^T\boldsymbol{A}\)</span>, a vector of length <span class="math notranslate nohighlight">\(n\)</span>. We have then, using the definition of the Jacobian,</p>
 <div class="math notranslate nohighlight">
 \[
 \alpha = \boldsymbol{z}^T\boldsymbol{x},
@@ -1366,8 +1366,9 @@ In order to find the derivative of <span class="math notranslate nohighlight">\(
 <p>which means that (using our previous example) we have</p>
 <div class="math notranslate nohighlight">
 \[
-\frac{\partial \alpha}{\partial \boldsymbol{x}} = \boldsymbol{z}^T=\boldsymbol{y}^T\boldsymbol{A}.
+\frac{\partial \alpha}{\partial \boldsymbol{x}} = \boldsymbol{z}=bm{A}^T\boldsymbol{y}.
 \]</div>
+<p>Note that the resulting vector elements are the same for <span class="math notranslate nohighlight">\(\boldsymbol{z}^T\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{z}\)</span>, the only difference is that one if just the transpose of the other.</p>
 <p>Since <span class="math notranslate nohighlight">\(\alpha\)</span> is a scalar we have <span class="math notranslate nohighlight">\(\alpha =\alpha^T=\boldsymbol{x}^T\boldsymbol{A}^T\boldsymbol{y}\)</span>. Defining now <span class="math notranslate nohighlight">\(\boldsymbol{z}=\boldsymbol{x}^T\boldsymbol{A}^T\)</span> we find that</p>
 <div class="math notranslate nohighlight">
 \[
@@ -1610,7 +1611,7 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9952495547161103
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9963864072152893
 </pre></div>
 </div>
 </div>
@@ -1627,7 +1628,7 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.00950269957492606
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.008435888964550838
 </pre></div>
 </div>
 </div>
@@ -1642,23 +1643,23 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.0193453  0.01242273 0.01256189 0.01871582 0.02840224 0.05004617
- 0.03536673 0.00706459 0.01326608 0.00817056 0.0443659  0.02090061
- 0.03724642 0.02376036 0.01943082 0.11352543 0.03437582 0.04426816
- 0.0055043  0.00779804 0.00415473 0.01761413 0.01206708 0.0234505
- 0.02928941 0.03893106 0.04678736 0.08370888 0.0739975  0.02035804
- 0.04028979 0.03739417 0.04670006 0.02383034 0.00416461 0.03232719
- 0.0156548  0.00393128 0.0957156  0.01406736 0.02551463 0.00388494
- 0.06432223 0.03509912 0.01422494 0.04787099 0.01323884 0.02803041
- 0.00075315 0.01037947 0.03255258 0.01631138 0.02129803 0.00715835
- 0.00427039 0.02298589 0.00128634 0.00877989 0.00444561 0.0459682
- 0.01751529 0.02917931 0.00791291 0.01207435 0.01960258 0.00431511
- 0.00407056 0.00220229 0.00267599 0.03371334 0.01654471 0.10880921
- 0.07557024 0.01944669 0.01542944 0.07143876 0.0136671  0.00695357
- 0.0021877  0.08084392 0.02862254 0.02701924 0.0499586  0.00202464
- 0.0113975  0.03423556 0.09652672 0.05150928 0.02979908 0.00569317
- 0.00112431 0.00780259 0.01200342 0.00397836 0.06287717 0.00403785
- 0.0068094  0.02086933 0.0165866  0.0066916 ]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.00168141 0.0025074  0.01497781 0.06168748 0.04101587 0.03079528
+ 0.0211496  0.04046799 0.00054272 0.01029589 0.02442398 0.04419696
+ 0.02566258 0.00868337 0.02649453 0.00147197 0.00546438 0.00233441
+ 0.01990155 0.00808144 0.00681644 0.07093644 0.01693    0.00965032
+ 0.00287154 0.00758025 0.02521597 0.00057106 0.01550656 0.04495095
+ 0.03995478 0.01056549 0.00820764 0.00089459 0.02082761 0.03583342
+ 0.01907538 0.00018425 0.02539993 0.02166269 0.0035417  0.0025125
+ 0.01742183 0.01586048 0.03791967 0.02344111 0.02195273 0.03500198
+ 0.01539    0.01176895 0.02868501 0.00201915 0.01588619 0.00754532
+ 0.01135107 0.01242803 0.06121384 0.01678286 0.05855607 0.02206138
+ 0.06031681 0.00523269 0.00911938 0.06038053 0.01957153 0.00449439
+ 0.00191249 0.01152107 0.02335522 0.04573105 0.02612167 0.010154
+ 0.00867698 0.0814721  0.01693278 0.01844381 0.00781035 0.01725808
+ 0.00645183 0.00054144 0.00452742 0.00406231 0.01619802 0.01073921
+ 0.00074389 0.07943621 0.01788423 0.04637311 0.0348171  0.00689391
+ 0.04087592 0.09631112 0.03634298 0.04516608 0.0183718  0.01817919
+ 0.07297557 0.00578738 0.00465403 0.00174508]
 </pre></div>
 </div>
 </div>
@@ -1727,15 +1728,15 @@ but now splitting the data into a training set and a test set.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.96769675 -0.05402365  6.44997121 -3.20038136  1.84614066]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.03739357 -0.54302039  7.21493572 -3.15550086  1.44651551]
 Training R2
-0.996694948530336
+0.9973102337301051
 Training MSE
-0.00740173378493795
+0.006158537606875468
 Test R2
-0.9920099299023244
+0.9954911579937146
 Test MSE
-0.013537225644261203
+0.010751622516868333
 </pre></div>
 </div>
 </div>
@@ -4147,24 +4148,29 @@ Similarly, <a class="reference external" href="https://arxiv.org/abs/1803.08823"
 </div>
 <div class="section" id="deriving-the-lasso-regression-equations">
 <h2>Deriving the  Lasso Regression Equations<a class="headerlink" href="#deriving-the-lasso-regression-equations" title="Permalink to this headline">¶</a></h2>
-<p>Using the matrix-vector expression for Lasso regression and dropping the parameter <span class="math notranslate nohighlight">\(1/n\)</span> in front of the standard means squared error equation, we have the following <strong>cost</strong> function</p>
+<p>Using the matrix-vector expression for Lasso regression, we have the following <strong>cost</strong> function</p>
 <div class="math notranslate nohighlight">
 \[
-C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
 \]</div>
 <p>Taking the derivative with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
 <div class="math notranslate nohighlight">
 \[\begin{split}
-\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 &amp; \beta &gt; 0 \\-1 &amp; \beta &lt; 0, \end{array}\right.
+\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 &amp; \beta &gt; 0 \\-1 &amp; \beta &lt; 0, \end{array}\right.
 \end{split}\]</div>
 <p>we have that the derivative of the cost function is</p>
 <div class="math notranslate nohighlight">
 \[
-\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
+\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
 \]</div>
 <p>and reordering we have</p>
 <div class="math notranslate nohighlight">
 \[
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\]</div>
+<p>We can redefine <span class="math notranslate nohighlight">\(\lambda\)</span> to absorb the constant <span class="math notranslate nohighlight">\(n/2\)</span> and we rewrite the last equation as</p>
+<div class="math notranslate nohighlight">
+\[
 \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
 \]</div>
 <p>This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a class="reference external" href="https://cvxopt.org/">CVXOPT</a>. We will discuss this later.</p>

doc/LectureNotes/_build/jupyter_execute/week35.py

@@ -231,7 +231,7 @@
 # We define the partial derivatives of the various components of $\boldsymbol{y}$ as functions of $x_i$ in terms of the so-called [Jacobian matrix](https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant)
 
 # $$
-# \boldsymbol{J}=\frac{\partial \boldsymbol{y}}{\partial \boldsymbol{x}}=\begin{bmatrix} \frac{\partial y_0}{\partial x_0} & \frac{\partial y_0}{\partial x_1} & \frac{\partial y_0}{\partial x_2} & \dots & \dots & \frac{\partial y_0}{\partial x_{n-1}} \\ \frac{\partial y_0}{\partial x_0} & \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} & \dots & \dots & \frac{\partial y_1}{\partial x_{n-1}} \\
+# \boldsymbol{J}=\frac{\partial \boldsymbol{y}}{\partial \boldsymbol{x}}=\begin{bmatrix} \frac{\partial y_0}{\partial x_0} & \frac{\partial y_0}{\partial x_1} & \frac{\partial y_0}{\partial x_2} & \dots & \dots & \frac{\partial y_0}{\partial x_{n-1}} \\ \frac{\partial y_1}{\partial x_0} & \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} & \dots & \dots & \frac{\partial y_1}{\partial x_{n-1}} \\
 # \frac{\partial y_2}{\partial x_0} & \frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2} & \dots & \dots & \frac{\partial y_2}{\partial x_{n-1}} \\
 # \dots & \dots & \dots & \dots & \dots & \dots \\
 # \dots & \dots & \dots & \dots & \dots & \dots \\
@@ -281,7 +281,7 @@
 
 # with $\boldsymbol{y}$ a vector of length $m$, $\boldsymbol{A}$ an $m\times n$ matrix and $\boldsymbol{x}$ a vector of length $n$. We assume also that $\boldsymbol{A}$ does not depend on any of the two vectors.
 # In order to find the derivative of $\alpha$ with respect to the two vectors, we define an intermediate vector $\boldsymbol{z}$. We define first
-# $\boldsymbol{z}^T=\boldsymbol{y}^T\boldsymbol{A}$, a vector of length $n$. We have then
+# $\boldsymbol{z}^T=\boldsymbol{y}^T\boldsymbol{A}$, a vector of length $n$. We have then, using the definition of the Jacobian,
 
 # $$
 # \alpha = \boldsymbol{z}^T\boldsymbol{x},
@@ -290,9 +290,11 @@
 # which means that (using our previous example) we have
 
 # $$
-# \frac{\partial \alpha}{\partial \boldsymbol{x}} = \boldsymbol{z}^T=\boldsymbol{y}^T\boldsymbol{A}.
+# \frac{\partial \alpha}{\partial \boldsymbol{x}} = \boldsymbol{z}=bm{A}^T\boldsymbol{y}.
 # $$
 
+# Note that the resulting vector elements are the same for $\boldsymbol{z}^T$ and $\boldsymbol{z}$, the only difference is that one if just the transpose of the other.
+# 
 # Since $\alpha$ is a scalar we have $\alpha =\alpha^T=\boldsymbol{x}^T\boldsymbol{A}^T\boldsymbol{y}$. Defining now $\boldsymbol{z}=\boldsymbol{x}^T\boldsymbol{A}^T$ we find that
 
 # $$
@@ -2604,26 +2606,32 @@ print(covariance_matrix)
 
 # ## Deriving the  Lasso Regression Equations
 # 
-# Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following **cost** function
+# Using the matrix-vector expression for Lasso regression, we have the following **cost** function
 
 # $$
-# C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+# C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
 # $$
 
 # Taking the derivative with respect to $\boldsymbol{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
 
 # $$
-# \frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
+# \frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
 # $$
 
 # we have that the derivative of the cost function is
 
 # $$
-# \frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
+# \frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
 # $$
 
 # and reordering we have
 
+# $$
+# \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+# $$
+
+# We can redefine $\lambda$ to absorb the constant $n/2$ and we rewrite the last equation as
+
 # $$
 # \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
 # $$