update

doc/LectureNotes/_build/html/chapter2.html

@@ -997,14 +997,8 @@ decomposition of the design matrix.</p>
 <p>which gives us, using the orthogonality of the matrices <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{V}\)</span>,,</p>
 <div class="math notranslate nohighlight">
 \[
-\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y},
+\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y},
 \]</div>
-<p>Note here that when we perform the  multiplication of the various matrices, the orthogonal vectors of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span></p>
-<div class="math notranslate nohighlight">
-\[
-\boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}],
-\]</div>
-<p>that belong to <span class="math notranslate nohighlight">\(i&gt;p-1\)</span>, result in only  zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to <span class="math notranslate nohighlight">\(i=p-1\)</span>. This corresponds also to the number of singular values (these are all non-zero).</p>
 <p>It means that the ordinary least square model (with the optimal parameters) <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span>, corresponds to an orthogonal transformation of the output (or target) vector <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> by the vectors of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span>.</p>
 </div>
 <div class="section" id="further-properties-important-for-our-analyses-later">
@@ -1210,10 +1204,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.02762405215108776
-3.9005135872087155
-[[0.81974332 2.54886137]
- [2.54886137 8.96601782]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.04279651270127165
+4.090014704675496
+[[0.77618224 2.31644071]
+ [2.31644071 8.00233155]]
 </pre></div>
 </div>
 </div>
@@ -1250,10 +1244,10 @@ a more brute force way. Here we scale the mean values for each column of the des
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08356774001062162
-1.415534701258823
-[[1.         0.53815559]
- [0.53815559 1.        ]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08322642264994606
+1.7484669886413582
+[[1.        0.6960326]
+ [0.6960326 1.       ]]
 </pre></div>
 </div>
 </div>
@@ -1283,30 +1277,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 0.99594988  1.80661365]
- [ 0.59616454  1.54671213]
- [-0.93676229 -3.41351287]
- [-0.2942772  -0.48952201]
- [ 0.31574634  1.86425056]
- [ 0.57888946  1.08188077]
- [ 0.379203    0.55808001]
- [ 0.55824107  2.14995486]
- [ 0.20609082  0.88908038]
- [-2.39924562 -5.99353748]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.84886051 -1.52171908]
+ [-0.91616374 -3.74126425]
+ [ 1.4399427   4.17240522]
+ [-0.993865   -2.87927126]
+ [ 0.51782322  0.94077605]
+ [ 0.80279641  3.93037171]
+ [ 0.53952479  0.67970864]
+ [ 1.40829683  5.9212905 ]
+ [ 0.03326365 -0.11962537]
+ [-1.98275836 -7.38267217]]
           0         1
-0  0.995950  1.806614
-1  0.596165  1.546712
-2 -0.936762 -3.413513
-3 -0.294277 -0.489522
-4  0.315746  1.864251
-5  0.578889  1.081881
-6  0.379203  0.558080
-7  0.558241  2.149955
-8  0.206091  0.889080
-9 -2.399246 -5.993537
-          0         1
-0  1.000000  0.970057
-1  0.970057  1.000000
+0 -0.848861 -1.521719
+1 -0.916164 -3.741264
+2  1.439943  4.172405
+3 -0.993865 -2.879271
+4  0.517823  0.940776
+5  0.802796  3.930372
+6  0.539525  0.679709
+7  1.408297  5.921291
+8  0.033264 -0.119625
+9 -1.982758 -7.382672
+         0        1
+0  1.00000  0.97202
+1  0.97202  1.00000
 </pre></div>
 </div>
 </div>
@@ -1363,37 +1357,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>     0         1         2         3         4         5         6         7   \
 0   0.0  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   
-1   0.0  0.071441  0.075840  0.068261  0.072824  0.077706  0.057587  0.061646   
-2   0.0  0.075840  0.081837  0.071552  0.076958  0.082846  0.060016  0.064609   
-3   0.0  0.068261  0.071552  0.068799  0.073134  0.077711  0.060288  0.064431   
-4   0.0  0.072824  0.076958  0.073134  0.078101  0.083399  0.063979  0.068619   
-5   0.0  0.077706  0.082846  0.077711  0.083399  0.089524  0.067854  0.073045   
-6   0.0  0.057587  0.060016  0.060288  0.063979  0.067854  0.054411  0.058088   
-7   0.0  0.061646  0.064609  0.064431  0.068619  0.073045  0.058088  0.062195   
-8   0.0  0.066064  0.069651  0.068920  0.073668  0.078719  0.062064  0.066651   
-9   0.0  0.070874  0.075192  0.073777  0.079158  0.084920  0.066356  0.071476   
-10  0.0  0.047953  0.049858  0.051662  0.054774  0.058035  0.047737  0.050910   
-11  0.0  0.051391  0.053678  0.055302  0.058816  0.062517  0.051043  0.054582   
-12  0.0  0.055153  0.057877  0.059278  0.063241  0.067437  0.054650  0.058596   
-13  0.0  0.059272  0.062498  0.063619  0.068087  0.072839  0.058585  0.062983   
-14  0.0  0.063781  0.067585  0.068358  0.073391  0.078771  0.062874  0.067775   
+1   0.0  0.088476  0.091031  0.086154  0.087387  0.088765  0.076329  0.077009   
+2   0.0  0.091031  0.094722  0.088119  0.090039  0.092126  0.077467  0.078609   
+3   0.0  0.086154  0.088119  0.089551  0.090358  0.091266  0.082786  0.083087   
+4   0.0  0.087387  0.090039  0.090358  0.091620  0.092998  0.083026  0.083648   
+5   0.0  0.088765  0.092126  0.091266  0.092998  0.094867  0.083320  0.084277   
+6   0.0  0.076329  0.077467  0.082786  0.083026  0.083320  0.078845  0.078710   
+7   0.0  0.077009  0.078609  0.083087  0.083648  0.084277  0.078710  0.078812   
+8   0.0  0.077822  0.079901  0.083495  0.084391  0.085368  0.078651  0.078999   
+9   0.0  0.078776  0.081357  0.084018  0.085264  0.086608  0.078672  0.079278   
+10  0.0  0.067141  0.067560  0.075017  0.074773  0.074552  0.073013  0.072523   
+11  0.0  0.067449  0.068197  0.074992  0.074986  0.075012  0.072651  0.072342   
+12  0.0  0.067864  0.068953  0.075061  0.075302  0.075586  0.072361  0.072240   
+13  0.0  0.068391  0.069838  0.075228  0.075728  0.076282  0.072147  0.072222   
+14  0.0  0.069036  0.070860  0.075498  0.076270  0.077108  0.072011  0.072290   
 
           8         9         10        11        12        13        14  
 0   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  
-1   0.066064  0.070874  0.047953  0.051391  0.055153  0.059272  0.063781  
-2   0.069651  0.075192  0.049858  0.053678  0.057877  0.062498  0.067585  
-3   0.068920  0.073777  0.051662  0.055302  0.059278  0.063619  0.068358  
-4   0.073668  0.079158  0.054774  0.058816  0.063241  0.068087  0.073391  
-5   0.078719  0.084920  0.058035  0.062517  0.067437  0.072839  0.078771  
-6   0.062064  0.066356  0.047737  0.051043  0.054650  0.058585  0.062874  
-7   0.066651  0.071476  0.050910  0.054582  0.058596  0.062983  0.067775  
-8   0.071641  0.077062  0.054340  0.058416  0.062881  0.067769  0.073120  
-9   0.077062  0.083151  0.058040  0.062563  0.067525  0.072970  0.078942  
-10  0.054340  0.058040  0.042708  0.045603  0.048762  0.052206  0.055958  
-11  0.058416  0.062563  0.045603  0.048818  0.052330  0.056165  0.060349  
-12  0.062881  0.067525  0.048762  0.052330  0.056233  0.060502  0.065168  
-13  0.067769  0.072970  0.052206  0.056165  0.060502  0.065252  0.070453  
-14  0.073120  0.078942  0.055958  0.060349  0.065168  0.070453  0.076248  
+1   0.077822  0.078776  0.067141  0.067449  0.067864  0.068391  0.069036  
+2   0.079901  0.081357  0.067560  0.068197  0.068953  0.069838  0.070860  
+3   0.083495  0.084018  0.075017  0.074992  0.075061  0.075228  0.075498  
+4   0.084391  0.085264  0.074773  0.074986  0.075302  0.075728  0.076270  
+5   0.085368  0.086608  0.074552  0.075012  0.075586  0.076282  0.077108  
+6   0.078651  0.078672  0.073013  0.072651  0.072361  0.072147  0.072011  
+7   0.078999  0.079278  0.072523  0.072342  0.072240  0.072222  0.072290  
+8   0.079443  0.079991  0.072084  0.072091  0.072184  0.072370  0.072652  
+9   0.079991  0.080821  0.071698  0.071901  0.072198  0.072598  0.073105  
+10  0.072084  0.071698  0.068724  0.068098  0.067527  0.067011  0.066553  
+11  0.072091  0.071901  0.068098  0.067615  0.067191  0.066828  0.066528  
+12  0.072184  0.072198  0.067527  0.067191  0.066919  0.066714  0.066580  
+13  0.072370  0.072598  0.067011  0.066828  0.066714  0.066675  0.066714  
+14  0.072652  0.073105  0.066553  0.066528  0.066580  0.066714  0.066934  
 </pre></div>
 </div>
 </div>
@@ -1878,7 +1872,7 @@ Training MSE for OLS
 3.0
 </pre></div>
 </div>
-<img alt="_images/chapter2_254_1.png" src="_images/chapter2_254_1.png" />
+<img alt="_images/chapter2_252_1.png" src="_images/chapter2_252_1.png" />
 </div>
 </div>
 <p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
@@ -2154,7 +2148,7 @@ Training MSE for OLS
 [ 0. -0.]
 </pre></div>
 </div>
-<img alt="_images/chapter2_256_1.png" src="_images/chapter2_256_1.png" />
+<img alt="_images/chapter2_254_1.png" src="_images/chapter2_254_1.png" />
 </div>
 </div>
 <p>We bring then back our exponential function example and study all
@@ -2257,7 +2251,7 @@ Test MSE OLS
 0.008675369724975977
 </pre></div>
 </div>
-<img alt="_images/chapter2_258_1.png" src="_images/chapter2_258_1.png" />
+<img alt="_images/chapter2_256_1.png" src="_images/chapter2_256_1.png" />
 </div>
 </div>
 <p>Both these example send a clear message. The addition of a
@@ -2684,7 +2678,7 @@ Test MSE OLS
 0.958228616652075
 </pre></div>
 </div>
-<img alt="_images/chapter2_324_1.png" src="_images/chapter2_324_1.png" />
+<img alt="_images/chapter2_322_1.png" src="_images/chapter2_322_1.png" />
 </div>
 </div>
 <p>How can we understand this?</p>

doc/LectureNotes/_build/jupyter_execute/chapter2.py

@@ -536,17 +536,9 @@ print(np.abs(C-B))
 # which gives us, using the orthogonality of the matrices $\boldsymbol{U}$ and $\boldsymbol{V}$,,
 
 # $$
-# \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y},
+# \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y},
 # $$
 
-# Note here that when we perform the  multiplication of the various matrices, the orthogonal vectors of the matrix $\boldsymbol{U}$
-
-# $$
-# \boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}],
-# $$
-
-# that belong to $i>p-1$, result in only  zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to $i=p-1$. This corresponds also to the number of singular values (these are all non-zero).
-# 
 # It means that the ordinary least square model (with the optimal parameters) $\boldsymbol{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\boldsymbol{y}$ by the vectors of the matrix $\boldsymbol{U}$.
 
 # ## Further properties (important for our analyses later)